Using biometric data intelligence for education management

ABSTRACT

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for developing cognitive and behavioral metrics associated with a user. In some implementations, a system obtains data from a writing implement, the data indicative of a user performing a task by writing with the writing implement against a receiving device. The system extracts features from the obtained data. The system determines metrics that reflect characteristics of the user, the metrics that reflect cognitive characteristics of the user and metrics that reflect behavioral characteristics of the user. Based on the extracted features and the determined metrics, the system generates a user profile for the user. Based on the features indicating characteristics of the writing behavior of the user, the system modifies the generated user profile as a performance of the user changes over time. The system provides recommendations to improve the performance of the user.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation application of and claims the benefitof priority to U.S. Application No. 17/357,070, filed on Jun. 24, 2021,the contents of which are hereby incorporated by reference.

BACKGROUND

Conventional test-taking methods typically focus on whether anindividual answers a question correctly or incorrectly. In theseconventional methods, a test can pose a question for an individual toanswer in a subjective response or through objectivity, by selectingamong one or more answers. However, these conventional test-takingmethods only evaluate whether an answer is correct and do not evaluateor focus on the individual taking the test.

SUMMARY

In some implementations, a computing system uses biometric data obtainedfrom a writing implement to develop cognitive metrics and behavioralmetrics associated with a user. In particular, the computing systeminstructs a user to perform a task with a writing implement. The taskcan be an academic task (e.g., a homework or exam problem, an essay orshort response), a professional task (e.g., a job application activity),or another type of task. For instance, when the task is an academictask, the computing system can instruct the user to, for example, solvean algebraic equation, write out a sentence, answer a readingcomprehension question, solve a logical game question, and the like. Theuser performs the task with a writing implement, such as a smart pen, asmart stylus, or some other form of a writing device. The user canperform the task by interacting with the writing implement, such aswriting with the writing implement on a receiving device, such as paper,smart paper, or a tablet screen. The user performs the task by writingnotes to identify what the task is asking, writing various methods toperform the task, writing out one or more potential solutions to thetask, and writing a solution identified and developed by the user, toname a few examples. The receiving device, the writing implement, orboth can communicate with the computing system. The computing systemobtains data from one or more of the receiving device or the writingimplement and determine, based on the obtained data, various metricscharacterizing the user’s performance of the task or characterizing theuser.

The computing system extracts features from the obtained biometric data.The features extracted from the obtained biometric data describe amanner in which the user performs the task with the writing implement.The biometric data can be used similar to a blueprint to identifycharacteristics of the user. These features can include spatialfeatures, temporal features, pressure features, textual features, tiltfeatures, or combinations thereof. The features can also include textualfeatures that are provided from optical character recognitiontechniques. Each of these extracted features indicates characteristicsof a writing behavior of the user while performing the task, e.g., howthe user used the writing implement perform the task. The spatialfeatures describe the spatial representations of the motion of thewriting implement while user performs the task (e.g., representations ofwords, letters, or numbers written by the user). The temporal featuresdescribe temporal elements of the handwriting path, such as speed andacceleration with which the writing implement was used while performingthe task. The pressure features denote an amount of pressure the userapplied to the receiving device with the writing implement whileperforming the task. The tilt features denote the tilt angle of thewriting implement while the user performed the task. The computingsystem may extract other features from the obtained data, such as userdata (e.g., temperature or heart rate of the user) or environmental data(e.g., audio or video images of an environment of the user, temperatureof the environment).

Based on the features extracted from the obtained biometric data, thecomputing system determines metrics that characterize the user. Themetrics can include, for example, cognitive metrics, behavioral metrics,or both. The cognitive and behavioral metrics can include syntheticmetrics that characterize (e.g., measure) aspects of the user’s learningand mental abilities. The synthetic metrics can be based on newlycreated metrics. For example, the computing system create othersynthetic metrics that are separate from the metrics created by thecognitive and behavioral metrics. These cognitive and behavioral metricscan be applied to, e.g., students performing academic tasks, medicalprofessionals, business professionals, and professionals applying to jobapplications, among others.

As will be further described below, the cognitive and behavioral metricscan include toolsets that characterize or diagnose a user using thewriting implement. For instance, in an educational environment, thedetermined cognitive and behavioral metrics indicate to a teacher howwell his/her students are learning. The cognitive and behavioral metricscan indicate strengths and weaknesses of each student. Generally,students have different strengths and weaknesses, and a benefit of thesystem with the writing implement is that these cognitive and behavioralmetrics can not only define how well a student is learning, but can alsoaid in generating individualized recommendations to improve learning ineach student. Thus, this system can provide specifically tailoredfeedback to each particular user, with the feedback having beendeveloped based on that user’s own set of cognitive and behavioralmetrics generated responsive to that user’s use of the writingimplement.

Returning to the education example, the cognitive and behavioral metricscan reflect cognitive and behavioral characteristics that can beanalyzed to help improve, e.g., optimize a learning environment for thestudent. The cognitive metrics include, for example, a number sensemetric, a memory visualization metric, a reasoning metric, and an applyconnections metric. The behavioral metrics include, for example, a testtaking skill metric, a grit metric, a concentration metric, and aquestion comprehension metric. The computing system can evaluate anddetermine values for each of these cognitive, behavioral metrics, andother synthetically created metrics for the student using the writingimplement by combining various extracted features.

In another environment, such as a professional environment, the userwrites with the writing implement in a job application. The determinedcognitive and behavioral metrics can indicate, to a recruiter or hiringmanager, characteristics that indicate whether the applicant would be agood fit for a job. In some cases, the applicant may only need to writea few sentences with the writing implement for the reviewer to decidewhether the applicant should move to the next round of the jobinterview. This may be the case when the reviewer is searching for anapplicant with grit and concentration as strong skill sets. In anotherenvironment, such as with the use of a psychometrics test, a user canwrite and be analyzed using an AI-based skillset evaluation, which canbe vastly enhanced by time data and cognitive and behavioral metrics.

The computing system can generate profiles of users that write with thewriting implement. For example, the computing system can generate aprofile for the user that includes values for each of the number sensemetric, the memory visualization metric, the reasoning metric, the applyconnections metric, the test taking skill metric, the grit metric, theconcentration metric, and the question comprehension metric. Thecomputing system may also associate other values with the generated userprofile, such as the extracted features, and the raw data from both thewriting implement and the receiving device.

In some implementations, the computing system stores profiles forvarious users in a database. When a user performs a task with thewriting implement, the computing system generates a profile for the userand performs identity verification of the user. The computing system cancompare the generated user profile to stored profiles for the varioususers and determine whether the current writer is a previouslyrecognized user based on a match between the generated user profile andone of the stored profiles. In this sense, the computing system canidentify the user through their cognitive and behavioral metrics, thecharacteristics of the user’s writing behavior, or both. If thecomputing system identifies a previously recognized user, the computingsystem can generate recommendations tailored to the user’s performance.Alternatively, if the computing system does not match the user to apreviously recognized user, the computing system can store the generateduser profile in the database as a new user.

The computing system offers benefits to users who perform tasks with thewriting implements. In some implementations, the computing system cangenerate recommendations tailored to the user based on the cognitive andbehavioral metrics of the user performing the task with the writingimplement. For example, the computing system may instruct the user tosolve an algebraic equation with the writing implement if the computingsystem determines that the user’s grit metric and test taking skillmetric are low. Moreover, and as will be further described below, thecomputing system may provide recommendations that help the user improvethe metrics defined in the user’s profile. These recommendations canprovide to a client device of the user or to a supervisor monitoringover the progress of the user. The computing system can store theprovided recommendations and compare them with subsequent determinedvalues of cognitive and behavioral metrics to determine if theserecommendations were in fact helpful.

In one general aspect, a method performed by a computing systemincludes: obtaining, by the computing system, data from a writingimplement, the data indicative of a user performing a task by writingwith the writing implement against a receiving device; extracting, bythe computing system, features from the obtained data, the featuresindicating characteristics of a writing behavior of the user while theuser performs the task with the writing implement; determining, by thecomputing system, based on at least some of the extracted features fromthe obtained data, metrics that reflect characteristics of the user, themetrics including cognitive metrics that reflect cognitivecharacteristics of the user, behavioral metrics that reflect behavioralcharacteristics of the user, or both; based on the extracted features,the determined metrics, or both, generating, by the computing system, auser profile for the user; based on the features indicatingcharacteristics of the writing behavior of the user, modifying, by thecomputing system, the generated user profile as a performance of theuser changes over time; and providing, by the computing system,recommendations to improve the performance of the user associated withthe generated user profile, the recommendations based on the determinedmetrics and the generated profile.

Other embodiments of these and other aspects of the disclosure includecorresponding systems, apparatus, and computer programs, configured toperform the actions of the methods, encoded on computer storage devices.A system of one or more computers can be so configured by virtue ofsoftware, firmware, hardware, or a combination of them installed on thesystem that in operation cause the system to perform the actions. One ormore computer programs can be so configured by virtue havinginstructions that, when executed by data processing apparatus, cause theapparatus to perform the actions.

The foregoing and other embodiments can each optionally include one ormore of the following features, alone or in combination. For example,one embodiment includes all the following features in combination.

In some implementations, the receiving device includes at least one ofpaper, smart paper, a tablet, or a computer screen.

In some implementations, obtaining the data from the writing implementincludes: obtaining, by the computing system, environmental data thatinclude one or both of (i) media illustrating recorded video or imagesof the user performing the task with the writing implement and anenvironment in which the user is performing the task with the writingimplement and (ii) audio data recorded in the environment indicative ofaudible noises while the user is performing the task with the writingimplement.

In some implementations, the method includes obtaining, by the computingsystem, one or more of: test question data that describe (i) one or morequestions posed to the user to be answered with the writing implementand receiving device, (ii) a correct answer to each of the one or morequestions, and (iii) an answer rubric, and input data that describe (i)data identifying the writing implement and (ii) credentials of the userwriting with the writing implement; and wherein the recommendations toimprove the performance of the user are further based on the testquestion data, the input data, or both.

In some implementations, extracting the features from the obtained dataincludes one or more of: extracting, by the computing system, spatialfeatures that describe spatial features of handwritten text produced bythe user with the writing implement while performing the task;extracting, by the computing system, temporal features that describetemporal elements of handwritten movements produced by the user with thewriting implement while performing the task; extracting, by thecomputing system, pressure features that describe a pressure recordedbetween a tip of the writing implement and the receiving device whilethe user performs the task; extracting, by the computing system, tiltfeatures that describe one or more tilting angles of the writingimplement while the user performs the task; and extracting, by thecomputing system, textual features that describe characteristics of thewritten letters and numbers with the writing implement, wherein thetextual features also comprise time stamps associated with each of thehandwritten text.

In some implementations, determining metrics based on the at least someof the extracted features includes generating the cognitive metrics,including one or more of: generating, by the computing system, a numbersense metric; generating, by the computing system, a memoryvisualization metric; generating, by the computing system, a reasoningmetric; and generating, by the computing system, an applicableconnections metric.

In some implementations, determining metrics based on the at least someof the extracted features further includes generating the behavioralmetrics, including one or more of: generating, by the computing system,a test taking skill metric; generating, by the computing system, a gritmetric; generating, by the computing system, a concentration metric; andgenerating, by the computing system, a question comprehension metric.

In some implementations, generating the user profile includesgenerating, by the computing system, a score associated with each of thenumber sense metric, the memory visualization metric, the reasoningmetric, the applicable connections metric, the test taking skill metric,the grit metric, the concentration metric, and the questioncomprehension metric.

In some implementations, providing the recommendations to improve theperformance of the user includes providing, to a computing deviceassociated with the user or a supervisor of the user, one or more of: anindication of whether a written response by the user is correct orincorrect, the written response associated with the user performing thetask; a grit metric; and one or more corrective actions to improveperformance of the user, the one or more corrective actions including atleast one of (i) instructions to perform additional tasks of a similartype to the task, (ii) instructions to reduce an amount of noise in anenvironment, (iii) medical information of the user, and (iv)instructions to change writing approach.

In some implementations, the method further includes comparing, by thecomputing system, the generated profile to stored profiles; andidentifying, by the computing system, the user based on a match betweenthe generated profile and one of the stored profiles.

In a general aspect, a method performed by a computing system includes:obtaining, by the computing system, data from a writing implement, thedata indicative of a user writing a series of characters with thewriting implement on a receiving device; extracting, by the computingsystem, features from the obtained data, the features indicatingcharacteristics of a writing behavior of the user while writing theseries of characters; comparing, by the computing system, acharacteristic dataset based on the extracted features to one or morestored characteristic datasets for the same series of characters; and inresponse to identifying a stored characteristic dataset that matches thecharacteristic dataset based on the extracted features, validating anidentity of the user.

Embodiments can include one or any combination of two or more of thefollowing features.

In some implementations, the receiving device includes at least one ofpaper, smart paper, a tablet, or a computer screen.

In some implementations, extracting the features from the obtained dataincludes one or more of: extracting, by the computing system, spatialfeatures that describe spatial features of handwritten text produced bythe user with the writing implement while performing the task;extracting, by the computing system, temporal features that describetemporal elements of handwritten movements produced by the user with thewriting implement while performing the task; extracting, by thecomputing system, pressure features that describe pressure recordedbetween a tip of the writing implement and the receiving device whilethe user performs the task; and extracting, by the computing system,tilt features that describe one or more tilting angles of the writingimplement while the user performs the task.

In some implementations, the method includes based on the extractedfeatures, measuring an amount of ink generated by the writing implementfor each character in the series of characters.

In some implementations, the method includes the characteristic datasetincludes data based on the measured amount of ink for each character.

In some implementations, comparing the characteristic dataset to the oneor more stored datasets includes comparing the amount of ink for eachcharacter written by the user to the one or more stored datasets using aheuristic analysis.

In some implementations, the method includes validating an identity ofthe user includes determining whether credentials of the user match toan identification of a user associated with the identified storeddataset.

In some implementations, the method includes, in response to determiningthat the credentials of the user match to the identification of the userassociated with the identified dataset determining, by the computingsystem, that the user is not cheating.

In some implementations, in response to determining not identifying astored characteristic dataset that matches the characteristic dataset:providing, by the computing system, an indication that the user writingthe statement is another user.

In some implementations, the method includes: providing, by thecomputing system, an indication to the user to write each letter of thealphabet in capital and lowercase and various numbers with the writingimplement; obtaining, by the computing system, data from each letter ornumber written by the user with the writing implement; and storing, bythe computing system, a template for each letter and number for the userin a dataset.

The details of one or more embodiments of the subject matter of thisspecification are set forth in the accompanying drawings and thedescription below. Other features, aspects, and advantages of thesubject matter will become apparent from the description, the drawings,and the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A is a block diagram that illustrates an example of a system fordetermining cognitive and behavioral metrics of an individual using awriting implement.

FIG. 1B is another block diagram that illustrates an example of a systemfor determining cognitive and behavioral metrics of an individual usinga writing implement.

FIG. 2 is an illustration of an example of input data obtained from awriting implement.

FIG. 3 is an illustration of an approach for generating feature datafrom input data obtained from a writing implement.

FIG. 4 is an illustration of example plots of grit score values versusexam score values.

FIGS. 5A-5C are illustrations of examples of anti-cheatingidentification.

FIGS. 6A-6B are illustrations of examples of plots showing individualheuristics for different letters for different users that use thewriting implement.

FIGS. 7 and 8 are flow diagrams.

Like reference numbers and designations in the various drawings indicatelike elements.

DETAILED DESCRIPTION

FIG. 1A is a block diagram that illustrates an example of a system 100for determining cognitive and behavioral metrics of a user 110 using awriting implement. The example of FIG. 1A shows a user writing with thewriting implement by performing a task in an educational environment,such as taking a pop quiz or test at school. However, the techniquesdescribed here can be applied to other systems as well, such as inprofessional environments where applicants apply for jobs and medicalenvironments for diagnosing the cognitive and behavioral abilities ofpatients. The system 100 includes a computing system 102 that has aconnected monitor or display 112. The user 110 writes with a writingimplement 108 against a receiving device 106. The computing system 102communicates with the receiving device 106, the writing implement 108,or both. FIG. 1A illustrates various operations in stages (A) through(I) which can be performed in the sequence indicated or in anothersequence.

The system 100 also includes a network 116 and a knowledge server 104.In some implementations, the network 116 can include a local network ora network connected over the Internet. For instance, the network 116 caninclude a Bluetooth, Wi-Fi, ZigBee, or another form of communicationconnection. The receiving device 106, the writing implement 108, or bothcommunicate with the computing system 102 over a local network ofnetwork 116. The computing system 102 communicates with the knowledgeserver 104 over a local network or over the Internet.

The computing system 102 can include a desktop, a laptop, a smarttablet, or a handheld device, to name a few examples. The writingimplement 108 can be, for example, a smart pen, a smart stylus, or someother form of a writing device that communicates with the computingsystem 102. The receiving device 106 can include, for example, paper,smart paper, a tablet screen, or a screen of a laptop, desktop, orhandheld device. The receiving device 106 can also include virtualcollaboration whiteboards and smart screens, which can track the user’swriting when linked with the writing implement 108.

As illustrated in system 100, the writing implement 108 can be a smartpen. The computing system 102 provides a task to the user 110, that theuser 110 can perform by writing with the writing implement 108 on thereceiving device 106. The task instructed by the computing system 102can include, for example, solving a mathematical equation, writing out asentence or a phrase, answering a reading comprehension question,solving a logical game, and performing other mathematical operations. Inother situations, the task can include filling out a job application,writing statements to be analyzed for medicinal purposes, filling outcontracts for government, military, and federal employees, and any otherform of task that involves writing.

In the case that the writing implement 108 corresponds to a smart pen,the smart pen includes the capability to write by communicating with thereceiving device 106. In particular, the receiving device 106 can recordpressure and/or movements received by the tip of the smart pen while thepen is in contact with the receiving device 106. In response, thereceiving device 106 can display digital ink in an amount and directionbased on the pressure and movements applied by the smart pen by theuser. For example, the user may be instructed with a task to write out aset of numbers, such as “345.” When the user writes the number “4” withthe smart pen on the receiving device 106, the receiving device 106displays to the user the number “4” as written by the user. The mannerin which the user writes the number “4,” such as the speed of the user’shand movement, the length of each stroke in the written number “4,” theamount of time taken to write the number “4,” among othercharacteristics of the user’s writing, are captured by the writingimplement 108,the receiving device 106, or both. These characteristicsare provided to the computing system 102, and ultimately, transmitted tothe knowledge server 104 where additional processes take place. In somecases, only the writing implement 108 captures these characteristics andcommunicates with the computing system 102.

The knowledge server 104 extracts various features from the dataprovided by the computing system 102. These various features can includespatial features, temporal features, pressure features, tilt features,and textual features, to name a few examples. As mentioned, each ofthese features can denote and describe characteristics of how the userused the writing implement to perform the instructed task. In someimplementations, each of these features can help define not onlycharacteristics of how the user wrote with the writing implement, butalso, characteristics of the user, such as the body temperature of theuser while the user writes and a heart rate of the user while the userwrites.

In some implementations, the knowledge server 104 determines metricsthat reflect characteristics, e.g., cognitive metrics reflectingcognitive characteristics of the user, behavioral metrics reflectingbehavioral characteristics of the user, or both. Said another way, byextracting out characteristics of the user’s writing style and quirks,the knowledge server 104 can extrapolate, from the writing features,cognitive and/or behavioral characteristics of the user. This will befurther discussed below. Additional discussion of cognitive andbehavioral metrics can be found in U.S. Pat. Publication No.2019/0347953, the contents of which are incorporated here by referencein their entirety.

Generally, the coordination between the writing implement 108, thereceiving device 106, the computing system 102, and the knowledge server104 rely on a method known as item response theory. Item response theoryis the idea that these components in system 100 can measurenon-performing or non-performance related metrics to predict performanceof a user. Typically, performance is measured by whether a task isperformed correctly or incorrectly. However, with item response theory,the components of system 100 can analyze non-performance relatedmetrics, e.g., extracted feature vectors, to predict the performance ofuser 110. For example, the knowledge server 104 can analyze the type ofwriting implement used, ink amount, writing length, the textualcharacteristics, and an amount of effort, among others, that the user isproviding to perform the task. From that perspective, the knowledgeserver 104 can measure the user’s performance not solely based onwhether the user correctly performs the task, but the techniques andcharacteristics applied by the user to perform the task. Additionally,in item response theory, the knowledge server 104 can apply neuralnetworks to measure non-performing or non-performance related metrics topredict performance of a user as the number of feature sets continue togrow and expand.

By measuring these cognitive and behavior abilities, the system 100 canuse the cognitive and behavioral metrics as a synthetic toolset to helpimprove the performance of the user 110. For example, the system 100 canprovide recommendations to the user 110 based on the generated cognitiveand behavioral metrics. If the knowledge server 104 determines that oneor more of the categories in the cognitive and behavioral metrics werelow, then the knowledge server 104 can generate recommendations tailoredto that user 110 to improve those low categories. In subsequentmeasurements of the user 110 with the writing implement 108, the goal isto see areas of improvement in the cognitive and behavioral metrics thatwere low in previous tasks. Thus, the system increases the performanceof the user 110 in performing various tasks by analyzing and tailoringlearning of the user 110. Additionally, by creating synthetic cognitiveand behavioral metrics for evaluating user performance, these systemsallow for a large library of cognitive and behavioral metrics to becreated. These metrics can act as proxies that can be used for other andfuture researchers’ projects.

During stage (A), the user 110 is instructed to perform a task. Thecomputing system 102 can provide a task to the user 110 through thereceiving device 106, in the case that the receiving device 106 is asmart tablet or a touch screen device. As illustrated in system 100, thetask includes two questions that the user 110 is requested to answer.The first question asks the user 110 the following question: “What isyour favorite color?” The second question asks the user 110 thefollowing question: “Solve for X: 10-X=7”. The receiving device 106provides the user 110 with space to write down notes to identify thetask, write various methods to perform the task, write one or morepotential solutions to each task, and write out the identified solutionto each task.

The user 110 answers the first question of the task using the writingimplement 108. The answer to the first question includes “John’sfavorite color is ...” and the phrase “Answer: Blue” circled. Then, theuser 110 answers the second question of the task using the writingimplement 108. As illustrated in example of system 100, the user 110writes out the following answer to solve the problem - adding “X” toboth sides of the equations to determine a new form of the equation as“10 = 7+X”. Then, using deduction, the user 110 can determine that thevalue of “X” is “3”. Consequently, the user 110 can select answer “C” inthe box at the bottom of the receiving device 106.

As the user 110 writes with the writing implement 108, the receivingdevice 106 displays the results of the writing. The results can includethe letters and numbers written by the user 110 with the writingimplement 108 to perform the task. Once the user 110 has finishedperforming the task, the user 110 may tap a notification on thereceiving device 106 or turn off the writing implement 108.

The writing implement 108, the receiving device 106, or both can provideraw data or biometric data to the computing system 102. The biometricdata can be used similar to a blueprint to identify characteristics ofthe user. In this case, the biometric data can correspond to how theuser 110 writes with the writing implement 108. The writing implement108 may include a camera that records media of the user 110 writing thesolution to the task and a microphone to record audio. The writingimplement 108, the receiving device 106, or both can provide the rawdata to the computing system 102 in real time. For example, the raw datacan include video or images recorded by the camera in the writingimplement 108, audio data recorded from the microphone of the writingimplement 108, positional data showing the movement of the writingimplement 108 as the user 110 performs the task, and an amount ofdigital ink generated by the receiving device 106 based on the pressureand movement applied by the writing implement 108. In some cases, thewriting implement 108 alone can also measure and record an amount ofdigital ink provided to the receiving device 106 based on its appliedpressure and movement. The raw data can also include other data sets,such as, written text, length of time for writing the letters andnumbers and speed at which the letters and numbers were written.

During stage (B), the computing system 102 obtains the raw data orbiometric data from the writing implement 108, the receiving device 106,or both. In some implementations, the computing system 102 can generatea package to provide to the knowledge server 104 over network 116. Thepackage can include test question data, the environmental data, and theinput data. In some implementations, the writing implement 108, thereceiving device 106, or both can provide the raw data directly to theknowledge server 104. In the case that the writing implement 108, thereceiving device 106, or both communicate directly with the knowledgeserver 104 and bypass the computing system 102, the computing system 102can provide the test question data and environmental data in a packageto the knowledge server 104. In some implementations, some data areobtained from other sources. For instance, the test question data can beobtained from a data storage hosted on the computing system 102 or adata storage associated with the knowledge server 104.

In some implementations, the test question data can include data thatdescribes the task to be performed. For example, the test question datacan include the two questions asked for the user 110 to answer-1)What isyour favorite color?; 2) Solve for X: 10-X = 7. The test question datacan also include the correct answers to each of those questions. In thecase that a question is a subjective question and no right answerexists, then the test question data can include an indication that thisis a subjective question. In some examples, e.g., if the question doesinclude a correct answer, the test question data can include the list ofpossible answers, the answer identified by the user 110, and the actualanswer to the question. The test question data can also include ananswer rubric that can be used to evaluate the specific steps andapproach taken to arrive at a particular answer to the problem.

In some implementations, the environmental data can include data thatillustrates an environment surrounding the user 110 as the user 110performs the task. For example, the environmental data can include videoand/or photo data captured by the writing implement 108 that shows thewriting of letters and numbers. Additionally, the video and photo datacan show the facial expressions of the user 110 as the user 110 performsthe task, which may aid in later developing the cognitive and behavioralmetrics of user 110. The environmental data can also include audio andvideo data recorded by the computing system 102, which may monitor theuser 110 and the environment surrounding the user 110. In this case, theaudio and video may record noises or visible movements in the backgroundof the user 110, such as TV noises, music, or one or more people movingbehind user 110. The environmental data can further include positionaldata to determine a location of user 110 and where user 110 isperforming the task. For example, the positional data can include GPScoordinates as well as data indicating the user 110 is located at home,at school, at a hospital, at a corporation, or in a coffee shop, to namea few examples.

In some implementations, the input data can include data provided by theuser 110 for responding to the task. For example, the input data caninclude credentials of user 110. The credentials can include ausername/email address and password of user 110 to verify that user 110is authenticated to use the computing system 102, the knowledge server104, and the writing implement 108. In some implementations, the user110 can provide the credentials to the computing system 102 beforeperforming the task with the writing implement 108 for authenticationpurposes.

The input data can also include data sources from external components.These external components can include input data from the keyboardassociated with the computing system 102, input data from the mouseassociated with the computing system 102, e.g., click data, pointerhovering, and pointer position in interface, touch screen dataassociated with the display 112, a pose of user 110 while performing thetask, and GPS data associated with a client device of the user 110. Thisdata can be added to the package 114 before transmitting.

The input data can also include identification data of the components insystem 100. For example, the identification data can include data thatrepresents the writing implement 108 and the represents the receivingdevice 106. This data can include an IP address, a MAC address, alabeled identifier, and a communication network ID, to name a fewexamples. The labeled identifier can indicate the type of device, forexample, the labeled data can indicate that the writing implement 108 isa “SMART PEN” and the receiving device 106 is a “TABLET.” Theidentification data can also include data that represents the computingsystem 102 that communicates with the writing implement 108 and thereceiving device 106. For example, the data representing the computingsystem 102 can include a hostname, an IP address, a MAC address, and thelike.

The input data can also include data obtained by the writing implement108 and the receiving device 106 while the user performs the task. Thedata obtained by the writing implement 108 can include written text,positional data of the writing implement 108, speed of the writingimplement 108, pressure amount of the writing implement 108, andtimestamps associated with each of these writing implement 108 datapoints. The data obtained by the receiving device 106 can include dataindicating amount of pressure received from the writing implement 108,an amount of digital ink generated that corresponds to the amount ofpressure received from the writing implement 108, and locational dataillustrating where the user wrote on the receiving device 106. In somecases, the writing implement 108 can measure and record each of theabove mentioned examples of input data without the use of the receivingdevice 106.

In some implementations, the computing system 102 can generate a package114 that includes the test question data, the environmental data, andthe input data. The computing system 102 can transmit the package 114 tothe knowledge server 104 over network 116. In some cases, the computingsystem 102 may store the package 114 in memory and send a notificationto the computing system 102 that includes the location in memory of thepackage 114 for retrieval of the data. Memory can be located on thecomputing system 102 or at another location, such as an externaldatabase.

The computing system 102 can generate and transmit a package 114 to theknowledge server 104 on a periodic basis. As the writing implement 108and the receiving device 106 obtain data, these devices can stream datain real-time or periodically to the computing system 102 or to theknowledge server 104. For example, the periodic basis can correspond toevery 30 seconds while the user 110 performs the task or one time afterthe user 108 has finished performing the task.

During stage (C), the knowledge server 104 can receive or retrieve thepackage 114 from the computing system 102. In some implementations, theknowledge server 104 extracts the test question data, the environmentaldata, and the input data from the package 114. The knowledge server 104can store the extracted test question data, environmental data, andinput data in memory for later retrieval. In some cases, the knowledgeserver 104 may store this extracted data in a historical database fortracking progress of a particular user, e.g., user 110, over time.

As illustrated in system 100, the knowledge server 104 provides theinput data to the feature extraction module 115. The feature extractionmodule 118 can extract from the input data a variety of features. Thefeatures can include, for example, spatial features, temporal features,pressure features, tilt features, and textual features. The spatialfeatures can correspond to pure spatial features of the text written byuser 110 with the writing implement 108. For example, these spatialfeatures can include a density of the handwriting, spaces betweendifferent words, spaces between letters in each word, handwriting size,and tilt angles of the letters. The handwriting size can include thelength of each word or letter in the vertical and horizontal directionas well as the width and height of each word or letter. These spatialfeatures can be measured in centimeters or another unit of measurementthat is set by a designer of the system 100.

The following list includes each of the spatial features that thefeature extraction module 118 can determine. For example, the featureextraction module 118 can determine an ink length and a cross out count.The cross out count can include a synthesized feature that correspondsto a count of the number of removed words or the number of scratched outwords. The feature extraction module 118 can also determine a word levellabel, an X and Y positions at the word level, a penInOutTime at theword level, a penDownTime at the word level, a penUpTime at the wordlevel, a word length at the word level, an average speed at the wordlevel, an X and Y positions at the letter level, a height of eachletter, a width of each letter, a letter length, a letter writing time,a letter average speed, a number of strokes for a character at theletter level, a letter size, a handwriting density, space between words,acceleration, and angles for letters.

The X and Y positions at the word level correspond to a position of theword on the receiving device 106. The penInOutTime at the word levelcorresponds to the penDownTime plus the penUpTime. The penDownTime atthe word level corresponds to the sum of all times taken when thewriting implement 108 was down writing the letters. The penUpTime at theword level corresponds to the sum of the time the writing implement 108was up away from the receiving device 106 while writing the word. Theword length at the word level corresponds to the summation of the totalink length of all letters per word. The average speed corresponds to theword length at the word level divided by the penInOutTime. The X and Ypositions at the letter level corresponds to the position for the letteron the receiving device 106. The letter length corresponds to the inklength of each letter. The letter writing time corresponds to thelongest time taken to write a letter minus the shortest time taken towrite a letter. The letter average speed corresponds to the letterlength divided by the letter writing time. The letter size correspondsto the letter height times the letter width. The handwriting density isa product of the mass of ink length per unit volume. The space betweenwords corresponds to the average space between each word. Theacceleration corresponds to the rate of change of the velocity betweenvarious letters and words. Lastly, the angles for letters correspond toa shape description for each written letter.

The temporal features can include measurements that represent thetemporal elements of a handwritten path. For example, these temporalelements can correspond to the speed taken by user 110 while writingwith the writing implement 108 to perform the task. In particular, thespeed can correspond to how fast user 110 writes each letter, how fastuser 110 writes each word, the speed at which the user 110 moves betweenwritten letters and written words, the acceleration at which the user110 writes with each word, and the acceleration at which the user 110moves between written letters and written words. The temporal elementscan also include the in-air time ratio of the user 110 with the writingimplement 108. The in-air time ratio can correspond to the amount oftime the user 110 holds the writing implement 108 in the air verse theamount of time the user 110 holds the writing implement 108 againstreceiving the receiving device 106. The in-air time ratio can indicatehow much time the user 110 spends writing on the receiving device 106against the amount of time the user 110 spends not writing, which can bean indication of whether the user does or does not understand how toperform the instructed task.

The following list includes each of the temporal features that thefeature extraction module 118 can determine. For example, the featureextraction module 118 can determine an initial time, a stroke time, astroke count, a gap time, a solve time, a total time, a delay featurestime a thinking time, an average speed, an initial speed, a terminationspeed, a pause time, a frequency of pauses, a solving order, and anumber of times revisiting the question.

The initial time can be considered as reading time, which can bedetermined as the time duration between a last stroke in the previousquestion until the first stroke in the current question. The initialtime is typically zero for the first question. The stroke timecorresponds to the time a user spent writing. The stroke countcorresponds to the sum of all strokes when answering the questions ofthe task. The gap time corresponds to a sum of all delays. The solvetime corresponds to the stroke time plus the thinking time. The totaltime corresponds to the solve time plus the initial time. The delay timefeatures corresponds to a delay of 1, 2, 3, 4, 5, 10, 20, or 30 seconds.The delay time features require adding all of the delays, including theactual values and the counts. The thinking time corresponds to an amountof time required for the user to think to perform the task (e.g., tosolve a problem), e.g., 5 seconds for example. The pause time can be avalue above 10 seconds, for example. The frequency of pauses cancorrespond to the value of pauses, such as 10 seconds or 30 seconds, toname a few examples. The solving order can correspond to the order inwhich the user answers the questions. The number of times revisiting thequestions corresponds to the number of times a user revisited aquestion, which may be helpful in determining the question comprehensionmetric.

The pressure features can include characteristics of the pressurerecorded between the tip of the writing implement 108 and the receivingdevice 106. Both the writing implement 108 and the receiving device 106may provide pressure data. The pressure data may be recorded in Pascals(PA), for example. The knowledge server 104 can extract various pressurerepresentations from the input data. For example, the various pressurerepresentations can include a change in pressure over time, the averageor mean pressure, the maximum mean pressure, and the speed of thepressure. The change in pressure over time can illustrate an amount ofpressure applied by the user 110 at different time units, such as 1 PAat 1 second, 2 PA at 2 seconds, 2 PA from 3 seconds through 20 seconds,and 1 PA at the 21-second mark. The average or mean pressure canindicate that the user 110 applied an average of 2 PA over the entireattempt. The speed of the pressure can correspond to a rate of changebetween the pressure amounts.

The tilt features can correspond to the characteristics of the tilt ofthe writing implement 108. For example, the tilt features can includethe mean or average tilt of the writing implement 108, the maximum tiltof the writing implement 108, a speed of the tilt of the writingimplement 108 in the X direction, and a speed of the tilt of the writingimplement 108 in the Y direction. The tilts can be measured from a frameof reference, such as at 0 degrees, the writing implement 108 iscompletely vertical and at 90 degrees, the writing implement 108 isparallel to the receiving device 106.

The textual features can correspond to the characteristics of thewritten letters and numbers with the writing implement 108. For example,the textual features can correspond to optical character recognition(OCR) results of the written letters and numbers. In someimplementations, the computing system 102 can perform the OCR functions.In other implementations, the knowledge server 104 can receive thecharacteristics of the written letters and numbers and perform OCR togenerate the textual features. The textual features can also include atime stamp indicating when the handwritten text was written with thewriting implement 108.

The feature extraction module 118 can also generate other features aswell. These other features can include, for example, score and points.The score can correspond to the maximum number of points for thequestions. The points can correspond to the points awarded to the userfor each question.

In some implementations, the feature extraction module 118 can generateeach of the spatial, temporal, pressure, textual, and tilt features fora set period of time. For example, if the amount of time taken by user110 to perform the task took five minutes, then the feature extractionmodule 118 can generate a set of features, e.g., spatial, temporal,pressure, and tilt features, every 10 seconds. In this example, thefeature extraction module 118 can generate 30 feature sets for theentire 5-minute period. In some examples, the feature extraction module118 can generate one set of features for the entire five-minute period.By generating periodic features over a set amount of time taken by theuser 110 to perform the task, the knowledge server 104 can analyze theuser 110’s writing behavior in chunks, and ultimately, monitor how theuser 110’s behavior and cognitive abilities evolve as the user 110performs the task.

During stage (D), the knowledge server 104 provides the cognitive module120 and the behavior module 122 with various inputs. These variousinputs can include the modalities of system 100. The modalities caninclude, for example, environmental data 107, the extracted featuresfrom the feature extraction module 118, and test question data 109 fromthe package 114. The modalities can be audio data, video data, the inputdata, and feature data. Additionally, the knowledge server 104 provideseach of the extracted features, e.g., spatial, temporal, pressure, tilt,and textual features, to both the cognitive module 120 and the behaviormodule 122. In some cases, the knowledge server 104 can provide a set ofeach of the extracted features to the cognitive module 120 on a periodicbasis. Continuing with the example from above, if the feature extractionmodule 118 generates 30 feature sets for the entire 5-minute period, theknowledge server 104 can provide one set from the 30 feature sets at atime. By combining and providing these modalities as input to both thecognitive module 120 and the behavior module 122, the cognitive andbehavioral metrics become more accurate and reliable.

In some implementations, the user 110 may interact with a user interfaceof the computing system 112 to perform the task instructed during stage(A) without the use of the writing implement 108. For example, the user110 may interact with a keyboard, a mouse, or a touchscreen of thedisplay 112 or the receiving device 106. In other examples, the user 110may write with a pen, a pencil, or another preferred form of writingthat does not have recording capabilities on the receiving device 106 asthe user 110 performs the task. In this case, the computing system 102can use different modalities, e.g., audio, video, movement data, totrack and monitor the user 110 as he/she performs the task. Inparticular, the computing system 102 can monitor the user 110 usingtechniques such as facial detection, object detection, classification,and other tracking techniques. In some implementations, the receivingdevice 106 can be the source of the input data when the user 110interacts with the receiving device 106. For example, the user 110 canwrite on a tablet display, e.g., receiving device 106, with a stylus,finger, or another device as a form of input. The computing system 102can then provide these recorded modalities along with the test questiondata and the input data without the use of the writing implement 108 tothe knowledge server 104 over network 116 to perform feature extractionand the continuing processes.

The knowledge server 104 can include a cognitive module 120 and abehavior module 122. The cognitive module 120 and the behavior module122 include a set of metrics that measure a performance of the user,such as user 110, and others. Both modules 120 and 122 can be used as atoolset to help measure, determine scores, and improve the cognitive andbehavioral abilities of user 110 using the determined scores. Forexample, the cognitive module 120 can determine a number sense metric, areasoning metric, a memory visual metric, and an apply / connectionsmetric. The behavior module 122, for example, can determine a testtaking skill metric, a grit metric, a concentration metric, and aquestion comprehension metric. The knowledge server 104 can indicatecognitive and behavioral abilities of the user 110 with each of thesemetrics.

In some cases, the knowledge server 104 can determine these metrics forthe user 110 just by the manner in which the user 110 writes with thewriting implement 108. Then, the knowledge server 104 can indicate tothe user 110 or to another individual supervising user 110 a performanceof the user 110 to help identify areas in which the user 110 is weak orneeds additional assistance. The benefit of system 100 is that, as willbe further described below, based on identifying areas in which the user110 has low metrics, the knowledge server 104 can generaterecommendations to improve the user 110 cognitive and behavioralavailabilities in these low metrics.

Both of the cognitive module 120 and the behavior module 122 candetermine the number of questions in the task, the number of questionsthe user 110 answered correctly, and the number of questions the useranswered incorrectly from the test question data 109 and the input data105. These modules (120 and 122) can use these results to furthercalculate their respective metrics, as discussed below.

During stage (E), the cognitive module 120 can determine metrics for theuser 110 based on the environmental data 107, the test question data109, and the extracted features from the feature extraction module 118.These metrics can include number sense, memory visualization, reasoning,and apply/connections. The number sense metric can correspond to auser’s flexibility of thoughts, flexibility in calculation, and anopenness of numbers. For example, the number sense metric can indicatehow adaptive is user 110 at performing tasks that involve looking atnumber problems from different points of view.

The memory visualization metric can indicate an ability of the user 110to mentally visualize portions of the task, to visualize patternrecognition, and to focus on memorization skills. The memoryvisualization metric can be measured by how quickly the user 110 is ableto correctly recall key concepts, formulas, patterns, and other problemsolving tactics. The memory visualization metric can also be determinedfrom reading comprehension problems, where the memory visualizationmetric can indicate how quickly a user can recognize a type of argument,the argument’s conclusion, and one or more other features, such as anargument’s assumption. The equation illustrated below corresponds to oneexample implementation of how the memorization visualization metric iscalculated:

$\begin{array}{l}{Memorization = {\sum{Memorization\mspace{6mu} weight \ast Correctness*}}} \\{\left( \frac{Average\mspace{6mu} resolve\mspace{6mu} time}{student\mspace{6mu} solvetime} \right) \ast} \\{\left( \frac{Question\mspace{6mu} Mark}{Total\mspace{6mu} Assessment\mspace{6mu} Mark} \right)OR\mspace{6mu} difficulty}\end{array}$

In equation 1, the memorization weight corresponds a value between 0 and1 or 0 and 100 for each specific question. The correctness correspondsto a 1 or 0 indicating whether the user answered the question correctly.The average resolve time corresponds to a time in seconds indicating howlong it takes the average person to solve the question. The studentsolve time corresponds to a time in seconds indicating how long it takesthe student to solve the question. The question mark can correspond to aweight value for the question and the total assessment mark cancorrespond to a weight value for each of the questions in theassessment. The difficulty can correspond to a value of difficulty ofthe question.

The cognitive module 120 can determine the reasoning metric. Thereasoning metric can measure an ability of the user 110 to criticallythink, execute logic on paper and even mentally, and executeproblem-solving skills. The reasoning metric can also measure an abilityof the user to develop responses in a creative manner to challengingproblems. For example, the reasoning metric can measure the ability ofuser 110 to think through the solution to each component of a particulartask in a logical manner. The equation illustrated below corresponds toone example implementation of how the reasoning metric is calculated:

$\begin{array}{l}{Reasoning\mspace{6mu} = {\sum{Reasoning\mspace{6mu} Weight \ast Correctness \ast}}} \\\left( \frac{Average\mspace{6mu} resolve\mspace{6mu} time}{student\mspace{6mu} solve\mspace{6mu} time} \right) \\{\ast \left( \frac{Question\mspace{6mu} Mark}{Total\mspace{6mu} Assessment\mspace{6mu} Mark} \right)OR\mspace{6mu} difficulty}\end{array}$

In equation 2, the reasoning weight corresponds a value between 0 and 1or 0 and 100 for each specific question. The correctness corresponds toa 1 or 0 indicating whether the user answered the question correctly.The average resolve time corresponds to a time in seconds indicating howlong it takes the average person to solve the question. The studentsolve time corresponds to a time in seconds indicating how long it takesthe student to solve the question. The question mark can correspond to aweight value for the question and the total assessment mark cancorrespond to a weight value for each of the questions in theassessment. The difficulty can correspond to a value of difficulty ofthe question.

The cognitive module 120 can determine an apply/connections metric. Theapply/connections metric can reveal user 110’s ability to connectabstract concepts and apply these connected abstract concepts to performthe task. The apply/connections metric can extend from conceptualunderstanding to applying association skills, which are reflected in themanner in which the user 110 writes with the writing implement 108. Theequation illustrated below corresponds to one example implementation ofhow the apply/connections metric is calculated:

$\begin{array}{l}{Apply/Connections} \\{= {\sum{Connections\mspace{6mu} Weight \ast Correctness \ast}}} \\\left( \frac{Average\mspace{6mu} resolve\mspace{6mu} Time}{student\mspace{6mu} solve\mspace{6mu} time} \right) \\{\ast \left( \frac{Question Mark}{Total\mspace{6mu} Assessment\mspace{6mu} Mark} \right)OR\mspace{6mu} difficulty}\end{array}$

In equation 3, the connections weight corresponds a value between 0 and1 or 0 and 100 for each specific question. The correctness correspondsto a 1 or 0 indicating whether the user answered the question correctly.The average resolve time corresponds to a time in seconds indicating howlong it takes the average person to solve the question. The studentsolve time corresponds to a time in seconds indicating how long it takesthe student to solve the question. The question mark can correspond to aweight value for the question and the total assessment mark cancorrespond to a weight value for each of the questions in theassessment. The difficulty can correspond to a value of difficulty ofthe question.

The cognitive module 120 can generate a score for each of these metrics.The score can correspond to a percentage between 0 and 100, or a numberbetween 0 and 1. The scores can be based on the number of questionsposed, the number of questions the user 110 answered correctly, thenumber of questions the user 110 answered incorrectly, and values fromthe one or more extracted features.

For example, the cognitive module 120 may determine the value of thenumber sense metric based an amount of time taken to perform the task,the average speed of the writing implement 108 by the user 110, and theperformance of the user 110. The performance can correspond to thenumber of questions answered quickly divided by the number of totalquestions in the task. As illustrated in system 100, two questions existand the user 110 has answered two questions correctly for a performancescore of 1 or 100%. If the user 110’s performance is high, average speedof writing is high, and the amount of time taken to perform the task islow, then the cognitive module 120 can determine the value of the numbersense metric to be a high number, such as 90%, for example. If the user110’s performance is low, average speed of writing is medium to low, andthe amount of time taken to perform the task is high, then the cognitivemodule 120 can determine the value of the number sense metric to be alow number, such as 20% for example.

The cognitive module 120 may determine the value of the memoryvisualization metric by determining a difference between a time whenuser 110 is first shown a problem of the task to a time when the user110 presses the writing implement 108 to the receiving device 106 toinitiate performing the task. The cognitive module 120 can thendetermine the memory visualization metric based on this difference valueand the performance value of the user 110. The difference value canindicate how well the user 110 has memorized a portion of the solutionor the solution itself. If the user has memorized the solution, then thedifference value will be very small. Thus, if the difference value issmall and the performance of the user is high, then the cognitive module120 can indicate that the memory visualization metric is high for user110.

The cognitive module 120 can determine the value of the reasoning metricbased on the total time taken to perform the task, the performance ofthe user 110, and an indication of the question that required reasoning.If the total time taken to perform the task is low and the performanceof the user 110 is high, then the cognitive module 120 can indicate thatthe user 110’s reasoning metric is high.

The cognitive module 120 can determine the value of theapply/connections metric based on a length of each stroke, the totaltime taken to perform the task, and the performance of the user 110. Ifthe total time taken to perform the task is low and the performance ofthe user 110 is high, then the cognitive module 120 can indicate thatthe user 110’s apply/connections metric is high. Moreover, if the lengthof each of stroke is longer, this may indicate that the user 110 isthinking about the answer because he is unsure of the answer. If thestroke length is short, this may indicate that the user is confident ofthe answer because the user desires to perform the task with haste.

In some implementations, the cognitive module 120 can indicate a valuefor each metric from the number sense, memory visualization, andapply/connections. Not only does the cognitive module 120 determinevalues for each of these metrics, but also the implication of thesevalues provides a measurement of the user 110’s cognitive abilities.Should user 110 have high reasoning and memory visualization metrics butlow number sense and apply/connections metrics, the knowledge server 104can provide recommendations to user 110 to improve their mathematicalstudies as their reading and vocabulary understanding is high. Otheranalyses can be determined and recommendations provided, as furtherdiscussed below.

During stage (F), the behavior module 122 can determine metrics for theuser 110 based on the environmental data 107, the test question data109, and the extracted features from the feature extraction module 118.These metrics can include test taking skills, grit, concentration, andquestion comprehension. The test taking skill can correspond to anability of a user to create strategies, understanding time management,and having self-awareness, each of which being measured while performingthe task. The ability to skillfully manage test taking time and handlethe pressures associated with the performing the task.

The grit metric can indicate a persistence and ability to deal withdifficult questions (or questions that the user 110 deems difficult)without giving up on performing the task. Moreover, the grit metric canindicate an ability to persevere through challenging and high-effortproblems on an assessment such as an instructed task. Generally, thegrit metric can indicate which users may perform the best in performingthe task because these users typically strive to persevere through tothe solution, whereas other users may give up. Grit may also indicate atendency to sustain interest in and effort towards your goal, e.g., theadage “whatever it takes to improve, can be used to define the gritmetric. In essence, how much a user persists through questions that aredeemed challenging by looking at a performance of the user based on, forexample, an amount of time taken to solve the most difficult questionsand the number of strokes taken to solve the problem. Likewise, userswith high grit scores can maintain a level of interest and efforttowards achieving a goal despite challenges and failures in the processof meeting that goal. Thus, a user with a high grit score may spend moretime and use more pen strokes on difficult questions.

The concentration metric can indicate an ability of a user to focuswithout taking numerous pauses through the assessment and respondingquickly to questions. The concentration metric can aid in determiningwhich users can focus on the task without being persuaded by othertasks. This metric also relates to positive metrics, where the user canfocus on problem solving instead of worrying about other externalfactors not related to performing the task.

The question comprehension metric can indicate an ability of a user tounderstand and communicate issues with self-control. Moreover, thequestion comprehension metric can indicate an ability to quickly andcorrectly understand the facts presented in a problem and formulate asolution to the instructed task. If a user is given a task with a longinstruction, the question comprehension metric can indicate how well theuser understood the long instruction based on the user’s solution andthe manner in which the user performed the task.

Similar to the cognitive module 120, the behavioral module 122 cangenerate a score for each of its metrics. Each score can correspond to apercentage between 0 and 100, or a number between 0 and 1. The score canbe based on the performance of the user (which can include the number ofquestions answered correctly and incorrectly), the number of questionsrequested to be answered, and values from the one or more extractedfeatures.

For example, the behavior module 122 can determine the value of the testtaking skills metric based on a speed rate of the user 110’s movement ofthe writing implement 108, an amount of time taken to perform the task,an amount of ink deposited by the writing implement 108 at the receivingdevice 106, and a performance of the user 110. If the behavior module122 determines that these values are high, then the behavior module 122can determine the test taking skills metric to be of a high value, e.g.,85%. A high test taking skills metric can indicate that the user 110 hasa strong tendency to create strategies, have self-awareness, andcomprehends time management while performing the task.

The behavioral module 122 can determine the value of the grit metricbased on a length of each stroke with the writing implement 108, anamount of time taken to perform each stroke with the writing implement108, a writing speed of the writing implement 108, a speed of thewriting implement 108 before the user performed the task, and theperformance of the user 110. The speed of the writing implement 108 cancorrespond to how quickly the user 110 moved the writing implement 108to the receiving device 106 upon reading the instructed task. A fastmovement may indicate that the user 110 has the desire and perseveranceto perform the task.

In some implementations, the behavior module 122 may determine the gritmetric using a particular equation. For example, the equation below(equation 4) may be the following:

$\begin{array}{l}{GritMetric = Perseverance \ast \frac{\# Questions Right}{Total\mspace{6mu} Questions} +} \\{Resilience \ast \frac{\# Questions\mspace{6mu} Wrong}{Total\mspace{6mu} Questions}}\end{array}$

In equation 4 above, the behavior module 122 can determine the gritmetric by multiplying a perseverance score by the performance of theuser 110. The performance of the user 110 can correspond to the numberof questions the user 110 answered correctly divided by the total numberof questions. The behavior module 122 may determine the perseverancescore of the user 110 by analyzing one or more extracted features, suchas the speed of the user 110’s movement with the writing implement 108,the change in speed of the user 110’s movement with the writingimplement 108 as the user 110 performs the task, and the performance ofthe user 110. The perseverance score is applied to the grit on thecorrect questions and resilience applies the same equation, equation 4,on questions that were incorrectly calculated. The resultant value ofthe perseverance score multiplied by the performance is added to aresilience score multiplied by a particular factor.

$Grit\mspace{6mu} Metric = {\sum\limits_{k = 1}^{n}{diff_{k} \ast correctness_{k} \ast \left( {stoke\mspace{6mu} count_{k} + total time_{k} + ink\mspace{6mu} lenght_{k}} \right)}}$

In equation 5 above, illustrates a different equation to calculate thegrit metric from equation 4. For example, the diff_(k) corresponds tothe difficulty level of a question k, such that each level has aparticular weight. For example, diff_(k) can be a value of 0.1 for aneasy question, 0.2 for a medium question, and 0.7 for a hard question.The diff_(k) can use other values as well. The correctness_(k) value cancorrespond to a Boolean value, either 0 or 1. In other examples, thecorrectness_(k) value can correspond to other values. The strokecount_(k) can correspond to the number of strokes performed in equationk. The total time_(k) can correspond to the total time spent to solve aquestion k. The ink length_(k) can correspond to all strokes which arecombined to form a continuous length for each question k. In someexamples, the grit score can rely on the calculation of a z-score usingthe combination of stroke count_(k), total time_(k), and ink length_(k).

The behavior module 122 can determine the resilience score and multiplythat value by the number of questions the user 110 answered incorrectlydivided by the total number of questions. The behavior module 122 candetermine the resilience score in a similar manner to how theperseverance score is determined, such as by analyzing the speed of theuser 110’s movement with the writing implement 108, the change in speedof the user 110’s movement with the writing implement 108 as the userperforms the task, the performance of the user 110, and an amount ofdigital ink deposited by the writing implement 108 on the receivingdevice 106. Thus, when the behavior module 122 adds the perseverancescore times the performance of the user 110 with the resilience scoretimes the number of questions the user 110 answered incorrectly dividedby the total number of questions, the behavior module 122 can generate agrit score.

The behavior module 122’s determination of the grit score is a strongindication of how well a user 110 will perform on tasks. For example, inan educational context and as will be illustrated below, students whoachieved high grit scores are typically students who received thehighest exam scores. Even when students answered questions incorrectly,these students did not give up and exerted more effort, these studentsappeared to answered more questions quickly overtime due to theirresilience and perseverance. The combination of perseverance andresilience creates a grit score that together can help identify whichusers are willing to put in the effort to perform the task.

In some implementations, the behavioral module 122 determines the valueof the concentration metric based on a stroke gap time, a count of longpauses, a writing speed, an initiation speed, and the performance of theuser 110. The stroke gap time can correspond to a total sum of time gapsbetween strokes made by the user 110 with the writing implement 108 whenwriting is not in progress. The time gap can correspond to a length oftime measured between each written letter in a word or between twoletters of two different words. If the stroke gap time is high, this mayindicate that much deliberation has occurred when performing the task.The count of long pauses can correspond to the total number of longdeliberations made by user 110 when performing the task. The count oflong pauses value may be high when the number of long deliberations issmall and the value is low when the number of the deliberations issmall. The writing speed can correspond to the speed of the movement ofthe writing implement 108 when the user 110 is performing the task. Thewriting speed can indicate not only the speed at instantaneous times butalso the difference in speed during times in which the user 110 isanswering each question or equation of the task. High values of writingspeed are typically correlated with high performance and confidence orconcentration in user 110’s performance of these tasks. The initiationspeed can correspond to the speed of the writing implement 108 when theuser 110 moved the writing implement 108 to the receiving device 106upon reading the instructed task. This initiation speed can indicate thespeed of understanding the question and the initiation of the questionanswering. A high initiation speed can indicate that the user 110 isfamiliar and confident about performing the task.

The behavioral module 122 can determine the value of the questioncomprehension metric based on an initial time to start performing thetask, a re-work indication, and the performance of the user 110. Theinitial time to start performing the task can indicate an amount of timetaken by user 110 to start writing with the writing implement 108 from atime at which the user 110 is shown the task. The re-work indication cancorrespond to whether the user has rewritten another solution to thesame task or whether the user has erased old work with the writingimplement 108 before selecting or entering an answer to a question ofthe task. For example, if the re-work indication is set, this canindicate that the user 110 may not comprehend the question being asked.The performance of the user 110 corresponds to the number of questionsthe user 110 answered right divided by the total number of questions inthe task.

In some implementations, the behavior module 122 provides a behavioralmeasurement of the user 110’s abilities. If the behavior module 122determines that the user 110 has a high grit and a high concentrationmetric with low test taking skills and question comprehension metrics,the knowledge server 104 can provide recommendations to the user 110 (ora supervisor of user 110) to improve their test taking skills. Thebehavior module 122 can indicate that the user 110 is willing to workhard in performing tasks but may not have enough experience in testtaking and comprehending questions, thus, the user 110 may just not morepractice in these question types.

During stage (G), the cognitive module 120 and the behavior module 122provide each of their metrics to the generate profile module 124. Thegenerate profile module 124 can generate a data structure that caninclude each of the metrics from the cognitive module 120, e.g., numbersense metric, memory visualization metric, reasoning metric, andapply/connections metric, and the behavior module 122, e.g., test takingskills metric, grit metric, concentration metric, and questioncomprehension metric. The generated user profile defines characteristicsof user 110 based on the metrics from the cognitive module 120 and thebehavior module 122. For example, the data structure can include atuple, a data struct, a vector, an array, and the like. The generateprofile module 124 can associate this data structure with thecredentials provided by the user 110, e.g., username and password, tothe computing system 102 before performing the task. The knowledgeserver 104 may use the credentials along with the stored metrics in thedata structure to identify and verify the authenticity of the user’scredentials. In some implementations, the data structure can includeother data, such as features that indicate characteristics of the user’swriting behavior, e.g., spatial features, temporal features, pressurefeatures, textual features, or tilt features. In some examples, the datastructure can include data indicative of a user’s writing behavior withrespect to a specific series of characters, as discussed further below.

For example, as illustrated in system 100, the generate profile module124 generates a user profile 111 of the user 110. The user profile 111can include the credentials of the user 110 along with each of themetrics from the cognitive module 120 and the behavior module 122.Collectively, the cognitive module 120 and the behavior module 122determined a grit metric of 75%, a number sense metric of 60%, areasoning metric of 30%, a memory visualization metric of 50%, anapply/connections metric of 80%, a test taking skills metric of 70%, aconcentration metric of 80%, and a question comprehension metric of100%.

During stage (H), the knowledge server 104 can provide the user profile111 to the comparison module 126. The comparison module 126 can comparedata from the user profile 111 to the data stored in the user profilesdatabase 130 to identify the user 110 writing with the writing implement108 and verify the authenticity of the user 110. The user profilesdatabase 130 can include one or more tables of data representingcredentials along with cognitive and behavioral metrics associated withprevious users. For example, the user profiles database 130 can includemultiple user profiles, each profile associated with a set of cognitiveand behavioral metrics, features indicative of characteristics of theuser’s writing behavior, or other data about the user.

In some cases, the user profiles database 130 includes multiple userprofiles for the same user. A user, such as user 110, may have 100 userprofiles in the user profiles database 130, for example. Each profilecan illustrate different instances when user 110 performed a task andthe resultant cognitive and behavioral metrics associated with therespective task. Each profile of the same user will contain the sameuser credentials. By having multiple user profiles, the knowledge server104 can monitor the learning of user 110. The learning can illustrate,for example, that the user 110’s grit has improved over time, the numbersense has remained the same, the memory visualization has increased, theapply/connections has decreased over time, the test taking skills haveincreased, the concentration has increased, and the questioncomprehension metric has decreased over time, to name some examples. Thelearning can also illustrate other patterns besides changes over time.This can include learning that shows the user 110, for example, has highgrit, concentration, and reasoning metrics when solving algebraicequations, but the user 110’s question comprehension, memoryvisualization, and apply/connection metrics are low when answeringreading comprehension questions. In other implementations, the userprofiles database 130 may store a single user profile for each user.Each user profile can then store the user credentials and the cognitiveand behavioral metrics associated with each task for that user over aperiod of time. By monitoring the learning of each user, the knowledgeserver 104 can track whether a user has improved or regressed hisscores, e.g., the cognitive and behavioral metrics, over time. Theknowledge server 104 can tailor recommendations to provide to the userdepending on whether the user has a learning setback, e.g., regression,or if the user is in fact improving his/her learning, for example.

In some implementations, the user profiles database 130 stores otherdata associated with each user profile. This other data can include thecorresponding input data, environment data, test question data, andextracted feature data. In this manner, the knowledge server 104 canmake other connections about the cognitive and behavioral metrics, suchas direct correlations between question types, e.g., math, verbal, andreading comprehension, with particular metrics. Additionally, theknowledge server 104 can determine, with the use of metrics, which areasthe users excel or are weak and require additional practice. Theknowledge server 104 can determine how users improve or declineassociated with particular question types.

The comparison module 126 can compare the user profile 111 to each ofthe stored user profiles in the user profiles database 130. For example,the comparison module 126 may first perform a search on the userprofiles and to search for any user profiles that have matchingcredentials to the credentials found in the user profile 111. If noprofiles exist that have matching credentials, then the knowledge server104 can indicate to the computing system 102 that this is the first timeuser 110 has used the writing implement 108, the receiving device 106,and the computing system 102. On other hand, if one or more profiles doexist that have matching credentials, then the comparison module 126 cantake an additional process to verify the user 110 is who they say theyare. This additional process can ensure that no user cheats by havinganother user perform a task that appears to be being performed by theuser associated with the supplied credentials.

The comparison module 126 can identify the one or user profiles thathave matching credentials to the credentials found in the user profile111. Then, the comparison module 126 can compare each metric found inthe user profile 111 to the metrics found in the identified one or moreuser profiles. To deem a match, the comparison module 126 can determinethe values must be within a predetermined value of one another. Thispredetermined value can be 5%, 10%, or 15%, depending upon a value setby an implementer of system 100. The lower the predetermined value, thehigher the accuracy of system 100. However, there is a chance that avery low predetermined value may make the system 100 inaccurate, in theevent that a user is misidentified. For example, if the predeterminedvalue is 1%, then the system 100 may not identify a user 110 with aprevious user profile, especially in the case that the user 101 has madedrastic improvements with their metrics over a period of time. Thus, thecomparison module 126 can also identify a generation date associatedwith a matching user profile to ensure the comparison module 126accurately identifies a particular user to previously generated userprofile. For example, if a generation date of a previous user profile iswithin a predetermined threshold of the current date when user profile111 is generated, and the metrics between the previous and current userprofile are within a threshold, the comparison module 126 can deem amatch exists.

During stage (I), if the comparison module 126 determines that metricsin the user profile 111 do not match to one or more identified userprofiles with the correct credentials, the knowledge server 104 canstore a new user profile in the user profiles database 130. The new userprofile can include the user 110’s credentials, data from the package114, e.g., the test question data, environmental data, and input data,and the metrics determined in the user profile 111. At a later point intime, the comparison module 126 can scan through the user profilesdatabase 130 and search for the newly inserted user profile.

In some implementations, the user profiles database 130 can make any ofthe values stored searchable. For example, if the knowledge server 104received a request to return all users with grit scores above 90%, theknowledge server 104 can perform a search in the user profiles database130 to return all user profiles with grit scores above 90%. Theseresults can include multiple user profiles associated with a singleuser, in the case that a user performed multiple tasks, and in eachtask, the knowledge server 104 determined that the user applied a gritof above 90%. Other parameters can be searched, such as question types,data associated with feature extraction, and device identification. Forexample, a user, such as a teacher, may request to search for all userswho used a particular writing implement and correlate those results withusers who performed similar tasks with a different writing implement.This may help identify to the teacher if a writing implement may not beworking correctly or whether the tasks are too difficult or easy acrossall users who use system 100.

The systems 100 and 101 may work in other environments different fromeducational environments. For example, the systems 100 and 101 may workin medical environments. In a medical environment, the user 110 may be apatient at a hospital. The computing system 102 can instruct the patientto write a particular phrase with the writing implement 108 on thereceiving device 106. The knowledge server 104 can perform stagessimilar to those defined in systems 100 and 101. Additionally, theknowledge server 104 can obtain heart rate data, skin temperature, andother medical readouts of user 110 from the writing implement 108 anduse this medical information along with the extracted features and thecognitive and behavioral metrics to make medical determinations.

For example, the knowledge server 104 may be able to use thisinformation to detect early stage Alzheimer’s detection, e.g., by way oflong pauses of writing words or odd writing patterns. Additionally, theknowledge server 104 may be able to detect other medical ailments suchas, Parkinson’s disease, e.g., by way of a shaky hand while writing, andcognitive aging. In some cases, these medical detections may bedetermined in not just a medical environment, but in normalenvironments, and the knowledge server 104 can indicate to a particularindividual, e.g., user 110, that they are exhibiting characteristicsassociated with early stage Alzheimer’s detection. Additionally, thesystems 100 and 101 may act as a possible lie detector by analyzing skintemperature, heart rate, and a detection that the user is writing in apattern that deviates from the user’s typical writing pattern.

FIG. 1B is another block diagram that illustrates an example of a system101 for determining cognitive and behavioral metrics of an individualusing a writing implement. The example of FIG. 1B is similar to FIG. 1A,includes similar components, but applied to different tasks. Similarcomponents between systems 100 and 101 will not be described again. FIG.1B illustrates various operations in stages (A) through (J) which can beperformed in the sequence indicated or in another sequence.

During stage (A), the user 110 is instructed to perform a task. Thecomputing system 102 can provide a task to the user 110 through thedisplay of the receiving device 106 and through the display 112 of thecomputing system 102. As illustrated in system 101, the task includestwo mathematical questions that the user 110 is instructed to answer.The first question asks the user 110 to answer the following questions:“How many letters in ‘THE QUICK BROWN FOX JUMPS OVER THE LAZY DOG’?” Thesecond question asks the user 110 the following question: “What is 100 x5?” The receiving device 106 can also provide the user 110 with space towrite notes to identify the task, write various methods to perform thetask, select a potential solution to each task, and write the identifiedsolution to each task.

As illustrated in system 101, the user 110 writes out the solution tothe first question with the writing implement 108 by counting the numberof letters in the phrase - “1, 2, 3 .... 26.” Additionally, the user 110circles the number 26 in his list to identify the answer to thequestion. Moreover, the user 110 circles “A) 26” to delineate the answerto the first question to the receiving device 106, the writing implement108, or both, and ultimately, to the knowledge server 104.

The user 110 writes out the solution to the second question with thewriting implement 108 by multiplying the number 100 by the number 5. Asshown in system 101, the user 110 is not able to come up with the answerand writes dots, indications of where to put the answer, and a questionmark to indicate that the user 110 is not able to answer the secondquestion. The receiving device 106, the writing implement 108, or both,and the computing system 102 captures this raw data. The user 110 canindicate that he has completed performing the task with two questions,as he provided an answer to the first question and did not provide ananswer to the second question.

During stage (B), the computing system 102 obtains raw data from thewriting implement 108 and the receiving device 106. The computing system102 can generate a package to provide to the knowledge server 104 overnetwork 116. The package 114, as illustrated in system 101, can includetest question data, the environmental data, and the input data. In someimplementations, the computing system 102 may generate a package 114 ona periodic basis as the user 110 is performing the instructed task andtransmit the package 114 to the knowledge server 104 on the periodicbasis. In some implementations, the computing system 102 may store thepackage 114 internally or externally in memory and send a location ofthe package 114 to the knowledge server 104 for retrieval.

During stage (C), the knowledge server 104 receives or retrieves thepackage 114 from the computing system 102. In some implementations, theknowledge server 104 can extract the test question data, theenvironmental data, and the input data from the package 114. Theknowledge server 104 can provide the extracted input data 105 from thepackage 114 to the feature extraction module 118. The feature extractionmodule 118 can extract a variety of features from the input data. Thesefeatures can include, for example, spatial features, temporal features,pressure features, textual features, and tilt features, associated withthe writing implement 108.

During stage (D), the knowledge server 104 provides the cognitive module120 and the behavior module 122 with various inputs. These variousinputs can include environmental data 107, test question data 109, andeach of the extracted features, e.g., spatial, temporal, pressure,textual, and tilt features. The test question data 109 can include datathat describes the task to be performed. The environmental data 107 caninclude data that illustrates an environment around the user 110 whilethe user 110 performs the task.

During stage (E), the cognitive module 120 generates cognitive metricsfor the user 110 using the environmental data 107, the test questiondata 109, and the extracted features from the feature extraction module118. These cognitive metrics can include number sense, memoryvisualization, reasoning, and apply/connections. The cognitive module120 can determine from the test question data 109 and the input data thenumber of questions associated with the task and whether the user 110correctly or incorrectly answered the questions in the task.

During stage (F), the behavior module 122 generates behavioral metricsfor the user 110 using the environmental data 107, the test questiondata 109, and the extracted features from the feature extraction module118. These behavioral metrics can include test taking skills, grit,concentration, and question comprehension. The behavioral module 122 canalso determine from the test question data 109 and the input data thenumber of correct and incorrect questions the user 110 answered from thetask.

During stage (G), the cognitive module 120 and the behavior module 122provide each of their determined metrics to the generate profile module124. The generate profile module 124 can generate a data structure thatcan include each of the metrics from the cognitive module 120, e.g.,number sense metric, memory visualization metric, reasoning metric, andapply/connections metric, and the behavior module 122, e.g., test takingskills metric, grit metric, concentration metric, and questioncomprehension metric. The generate profile module 124 can also includethe credentials of user 110 with this generated data structure. Thegenerated data structure or generated user profile 111 is associatedwith the user 110. Moreover, the generated data structure or generateduser profile 111 defines characteristics of the user 110. For example,the generated user profile 111 can include a grit metric of 10%, anumber sense metric of 8%, a reasoning metric of 80%, a memoryvisualization metric of 30%, an apply/connections metric of 10%, a testtaking skills metric of 30%, a concentration metric of 20%, and aquestion comprehension metric of 80%. These numbers are indicative ofthe user 110 performing the instructed task. For example, the user 110was not able to figure out question 2, and consequently, the cognitivemodule 120 determine the number sense metric, the concentration metric,and the apply/connections metric to each be very low.

During stage (H), the knowledge server 104 provides the generated userprofile 111 to the comparison module 126. The comparison module 126 cancompare the data from the generated user profile 111 to the data storedin the user profiles database to identify the user 110 and verify theauthenticity of the user 110. As illustrated in system 101, thecomparison module 126 identified the user 110 to be Alex based on thecomparison of the user 110’s supplied credentials and the comparison ofthe generated metrics in user profile 111 to various metrics in the userprofiles database 130. The comparison module 126 can retrieve one ormore user profiles 113 associated with Alex that show various metricsfor each task performed by Alex, e.g., user 110. By identifying the user110 as being Alex, the knowledge server 104 can determine with certaintythat the user 110 is in fact Alex and not another user pretending to beAlex.

During stage (I), the knowledge server 104 generates one or morerecommendations to provide to Alex, e.g., user 110, to improve Alex’sperformance. For example, as illustrated in system 101, the knowledgeserver 104 generates recommendations 131 that include “Practice Mathproblems,” “Do Math in Your Head,” and “Write out your work.” Theknowledge server 104 can analyze one or more factors of the generatedprofile to determine the types of recommendations to provide to the user110. For example, the knowledge server 104 can determine that the numbersense metric, the apply/connections metric, and the concentration metricare allow below a minimum threshold of 50%. If the knowledge server 104determines that any of these metrics are below a minimum threshold, theknowledge server 104 can obtain a set of recommendations that have beenknown to address and improve the user’s 110 performance with thesemetrics. For example, the knowledge server 104 can instruct the user toperform additional practice math problems and to practice performingmath problems in their head in seeing that the number sense metric iswell below the threshold. Additionally, if the knowledge server 104determines that the question comprehension metric and the memoryvisualization metrics were below a threshold, the knowledge server 104can determine that the user should read more books and study using flashcards to improve these features.

In some implementations, if the knowledge server 104 determines that theenvironment the user 110 is too noisy or loud, the knowledge server 104recommends the user 110 take actions to reduce the noise. These actionscan include, for example, moving to a different location, such as in adifferent room of the home, turning down or off the television volume,telling others around you to be quiet, adjusting the brightness of thelights, and turning down or off the music. Additionally, the knowledgeserver 104 can make these recommendations if the user 110 has a lowperformance and/or at least one of the cognitive and behavioral metricsare low. The goal is that reducing the noise in the environment willultimately help user 110 perform better.

In some implementations, if the user 110 is working an environment witha supervisor, the knowledge server 104 generates recommendations toprovide to the supervisor. For example, a supervisor may log into thecomputing system 102 using his or her credentials and indicate to thecomputing system 102 that she will be monitoring multiple users with thewriting implement 108 and the receiving device 106. The supervisor mayindicate to the computing system 102 the users he or she is monitoringand the credentials for each of those monitored users. In that manner,when the computing system 102 transmits the package 114 to the knowledgeserver 104, the knowledge server 104 can process the data and generaterecommendations to provide to a client device of the supervisor. Theknowledge server 104 can indicate that user A needs to practice mathproblems, user B needs to practice reading comprehension problems, anduser C needs to practice logical problems to improve their grit score,to name a few examples. The knowledge server 104 can also generate otherrecommendations. The other recommendations can indicate to the user,student, or supervisor that the user should change his/her writingapproach. This would require writing more slowly and more carefully toarrive at a correct answer.

In some implementations, the knowledge server 104 analyzes a variety ofuser profiles for a particular user to determine a user’s performanceover a period of time. For example, the knowledge server 104 candetermine how a user’s grit score and concentration metrics haveincreased over the past month. If the knowledge server 104 determinesthat these metrics have not increased at an acceptable rate, such as a2% increase for each task, then the knowledge server 104 can generateadditional recommendations to focus the user’s attention to performingadditional practice problems that help improve these metrics.

Additionally, the knowledge server 104 can indicate to the supervisorthe actual determined metrics of a particular user. For example, theknowledge server 104 can indicate that the user 110’s grit metric hasdecreased over a period of time or that the user 110’s concentrationmetric has increased over a period of time. This display of data, e.g.,such as through a graph or plot, can indicate to the supervisor areas ofimprovement that need to be applied to a particular user and areas thatthe user has improved. The knowledge server 104 can also display how themetrics for a particular monitored user or monitored multiple users havechanged over a period of time. This information can be important to aidin improving the performance of a user and of improving performances ofmultiple users.

In some implementations, the knowledge server 104 can monitor and trackmultiple users over time. As the users use the components shown insystems 100 and 101, the systems 100 and 101 can track the performanceof the users, e.g., whether the users have improved or declined in theircognitive and behavioral abilities, over time. In particular, theknowledge server 104 can monitor and track the performance of each userover time to see how the users’ performance changes or remains the sameover time.

Incrementally, as the knowledge server 104 generates cognitive andbehavioral metrics for each user, e.g., as the user performs a task, theknowledge server 104 can generate and provide tailored recommendationsto each of the students with the goal of improving their cognitive andbehavioral metrics. For example, if the knowledge server 104 determinesthat the grit score of user 110 is increasing by two points per month,the knowledge server 104 can continue providing tailored recommendationsto user 110 that maintain or improve his two-point increase per month.Alternatively, if the knowledge server 104 determines that user 110’sgrit score is decreasing by three points per month, then the knowledgeserver 104 can adjust its recommendations because the past tailoredrecommendations were not effective for that particular user as the user110 is regressing.

In some implementations, the knowledge server 104 can update each user’sprofile as each user performs additional tasks. In particular, theknowledge server 104 can update the cognitive and behavioral metrics ofthe user associated with that user’s profile, and perform this processfor each user profile. For example, the knowledge server 104 mayrecalculate each of the cognitive and behavioral metrics based on amoving average each time the user performs a particular task. In anotherexample, the knowledge server 104 can recalculate the cognitive andbehavioral metrics on a periodic basis, e.g., weekly, monthly, annually,based on a moving average of the values of the metrics. In anotherexample, the knowledge server 104 can erase the previously determinedcognitive and behavioral metrics each time new metrics are calculated sothat new metrics are provided each time a user performs a task.Alternatively, the knowledge server can store previous metrics so thatthe user or a supervisor can view progress of a monitored user, e.g.,user 110, over time to determine if the monitored user’s progress isslowing, accelerating, or remaining constant. Based on the monitoring ofthe users and their progress, the knowledge server 104 can generatetailored recommendations for each of the users.

This process of generating tailored recommendations, monitoring users’progress with the cognitive and behavioral metrics, and either refiningor regenerating new tailored recommendations is an incremental processthat continues over a users’ learning period. As the user 110 continuesto use system 100, the knowledge server 104 can learn whichrecommendations work best for a particular user by monitoring theresultant cognitive and behavioral metrics that are created. Forexample, the knowledge server 104 can generate tailored recommendationsfor a user over time that focus on improving one, a subset of, or all ofthe defined cognitive and behavioral metrics.

In some implementations, a supervisor can indicate to the knowledgeserver 104 which of the cognitive and behavioral metrics that theydesire a user to improve. The knowledge server 104 can then generatetailored recommendations that focus specifically on improving thecognitive and behavioral metrics designated by the supervisor. If theknowledge server 104 determines that the recommendations do not appearto be improving the user’s cognitive or behavioral metrics over a periodof time, then the knowledge server 104 may notify the supervisor of theuser, e.g., a parent, a teacher, a boss, a medical worker, or the userthemselves, that learning is not taking place.

During stage (J), the knowledge server 104 transmits the generatedrecommendations 131. For example, the knowledge server 104 can transmitthe generated recommendations 131 to the computing system 102, where thecomputing system 102 displays the generated recommendations 131 on thedisplay 112. As illustrated in system 101, the computing system 102 candisplay the generated recommendations of “Practice Math Problems,” “DoMath in Head,” and “Write Out Work.” These recommendations or correctiveactions can be provided to the user 110 after the user 110 has completedthe instructed task. The recommendations can also include an indicationof which questions the user 110 answered correctly and incorrectly.

Additionally, the knowledge server 104 can provide the recommendationsto a client device of the supervisor monitoring the user 110. In someimplementations, the knowledge server 104 can transmit a notification tothe client device of the supervisor to indicate to the user thatperformance metrics have been evaluated and recommendations have beengenerated. The supervisor can log into a web site hosted by theknowledge server 104 on the cloud network, for example, and thesupervisor can access the user profiles for each user the supervisor ismonitoring and the recommendations generated for each user. Afterwards,the supervisor can take actions based on the recommendations to tailorthe performance improvement for each user and monitor subsequentgenerated metrics of the user to determine if the actions andrecommendations were correct. This process is continuous and can beapplied to a user’s performance.

In some implementations, the knowledge server 104 employs one or moremachine learning techniques to any of the stages (C) through (J). Forexample, the one or more machine learning techniques may be used toautomatically extract features from the input data, generate cognitiveand behavioral metrics, generate a user profile, and generaterecommendations to provide for the user 110. Moreover, the knowledgeserver 104 may be able to provide the raw input data into the machinelearning techniques, and the machine learning techniques may be able tooutput structured data, e.g., recommendations to provide to the user 110to improve their performance. The machine learning techniques may employautomated correlations and algorithms to speed up the process ofgenerating cognitive/behavioral metrics and recommendations.

FIG. 2 is an illustration of an example of input data 200 obtained froma writing implement. For example, the input data 200 may be obtained bythe computing system 102 and the knowledge server 104 from the writingimplement 108. The writing implement 108 can provide this data in realtime to the computing system 102, the knowledge server 104, or to both.

A plurality of strokes, e.g., stroke 1, stroke 2, stroke 3, throughstroke N, may be performed on a particular page, e.g., receiving device106. FIG. 2 illustrates that a user may write on a variety of pagesusing the writing implement 108 for performing a task with one or morequestions. For example, on each page, a stroke may be identified by acontinuous trajectory from a point where the tip of the writingimplement 108 is in contact with the receiving device 106 to a pointwhere the writing implement 108 has lifted contact from the receivingdevice 106. Coordinates of the writing implement 108 may specify thestrokes during this continuous trajectory at identified time intervals,such as between 0.012 to 0.015 seconds. For example, as shown in system200, in Stroke 1 on Page:0.329.3244.0.58, the user 110 wrote withwriting implement 108 starting at the initial point of (051.94, 053.70)in X, Y coordinates. Moreover, the same Stroke 1 on the same page has awriting pressure of 70 and was provided on Oct. 26, 2017 at11:13:52:624. The time may be in 24-hour mode, so this time illustratedcorresponds to 11:13:52:624 in the morning. Each stroke can correspondto a portion of a letter, the letter itself, and portion of a word ifthe user 110 writes in a continuous trajectory manner.

FIG. 3 is a block diagram that illustrates an example of a system 300for generating feature data from input data obtained from a writingimplement. As illustrated in system 300, the user 110 has written thealgebraic equation of “4a + b”. The knowledge server 104 can analyze theraw data and extract features from the written representation of “4a +b”.

For example, the knowledge server 104 can consider raw input as pointsin the two-dimensional Euclidean plane from the collected input dataextracted from the package 114. The points in the two-dimensionalEuclidean plane can correspond to stroke positions and time stamps. Theknowledge server 104, and in particular, the feature extraction module118 can apply a Cartesian geometric formula to calculate the distanceand speed of the user 110’s movement with the writing implement 108,thereby creating feature values to be used by the cognitive module 120and the behavior module 122. As an example, the feature extractionmodule 118 can calculate distance and speed using the two stroke points(x1, y1) and (x2, y2) shown in system 300.

The distance of a stroke may be calculated between the two stroke points(x1, y1) and (x2, y2) using the following equation (6):

$lenght = \sqrt{\left( {x_{2} - x_{1}} \right)^{2} + \left( {y_{2} - y_{1}} \right)^{2}}$

The feature extraction module 118 can calculate the time between twostroke points using the following equation (7):

time duration  = t₂ − t₁

The feature extraction module 118 can determine the speed of the user110’s movement with the writing implement 108 by performing thefollowing equation (8):

$speed = \frac{\sqrt{\left( {x_{2} - x_{1}} \right)^{2} + \left( {y_{2} - y_{1}} \right)^{2}}}{t_{2} - t_{1}}$

In some implementations, the feature extraction module 118 can determineother features as well. For example, the feature extraction module 118can determine delay time by the difference in time stamps between thelast stroke of one character and the first stroke of a subsequentcharacter. The feature extraction module 118 can determine count ofpauses in the raw input of specific durations by counting the number oftime intervals during which no stroke is inputted. The featureextraction module 118 can determine the length of input by summing thelength of the strokes of all characters inputted by the user.Additionally, the feature extraction module 118 can determine whetherthe user 110 is performing rework by determining a movement exists withthe writing implement 108 in the negative x-axis and/or negative y-axisdirection.

The feature extraction module 118 can also calculate the number of allstrokes, e.g., total stroke, extractable from input data and calculate atotal time of use of the writing implement 108 by the user, e.g., totaltime. A total time of use of the writing implement 108 by the user 110for at least one question of a task may include a time when the user 110reads and deliberates on the at least one question after it ispresented, e.g., introduction time, and a time when the question isactually answered with the writing implement 108, e.g., solving time.The feature extraction module 118 can also calculate the time when theuser 110 reads and deliberates on the question, e.g., introduction time,and the time when the question is actually answered, e.g., solving time.Moreover, the time from a writing implement 108 being pulled away fromthe receiving device 106 of the last stroke of the answer to theprevious question to a writing implement 108 contacting the receivingdevice 106 for the first stroke of the answer to the next question maybe defined as the deliberation time, e.g., introduction time.

In some implementations, the feature extraction module 118 trackscoordinates of points forming each stroke. If the coordinate values ofthe points stay substantially the same for a predetermined time period,the feature extraction module 118 can determine that a timing delay hasoccurred. In addition, the feature extraction module 118 may calculate atotal time of delays and the number of delays in associated with atleast one question from the task for user 110. The predetermined timeperiod can be for example, one second, two seconds, three seconds, etc.Different weights may be applied depending on the length of thepredetermined time. Thus, the feature extraction module 118 cancalculate a total time and the number of delays.

With analysis of the raw data, the feature extraction module 118 candetermine with at least one question, stroke-drawing speed, e.g., strokevelocity in cm/second, initial speed of a stroke, ending speed, averagespeed, solving speed, which is the number of strokes per questiondivided by a solving time per question. Additionally, the featureextraction module 118 can calculate the sum of total length of strokes,e.g., ink length, area used for answering the question, complexity indirections of solving progression, e.g., entropy, cross-out, number ofproblems attempted, number of changes in question solving order, timetaken to start answering the next question when the order of questionanswering has changed, e.g., out of order time.

For example, the area used for performing a task, e.g., area, may becalculated to be the area where strokes are present. Complexity indirections of solving progression, e.g., entropy, may be calculated bydetermining that a stroke moving from left to right or from top tobottom is of low entropy and, in contrast, a stroke going from right toleft or from bottom to top is of high entropy. Cross-out may be tracedby dividing the result into cross-out of a number or a word, such as atypo cross out, crossing out part of a problem solving process, andcrossing out of the entire solving or an area corresponding. The problemattempted may be calculated to be the number of times when a question isattempted over a particular time period. The number of changes inquestion answering order, e.g., out of order variable, may be determinedby tracking when the user 110 skips a question to answer anotherquestion.

FIG. 4 is an illustration 400 of example plots of grit score valuesversus exam score values. As previously mentioned, a strong correlationexists between a grit score of a particular user and an exam score, inan educational context. Moreover, the grit score is a clear indicationof a user’s performance, as illustrated by the plot shown inillustration 400. For example, the plot has been divided into threedifferent sections to illustrate users with different grit scores-a lowgrit score 402, a medium grit score 404, and a high grit score 406. Eachof these grit scores are associated with values shown in the boxes. Gritscores with 2.1 and below are low grit scores; scores with valuesbetween 2.1 and 5.6 are medium grit scores; and, scores with values of8.8 and above are high grit scores. Likewise, this plot indicates thatthe higher the grit score corresponds to a higher exam score.

In some implementations, the knowledge server 104 can calculate the gritscore for a student compare grit score to previous grit score for thatsame user over time. The knowledge server 104 can identify a patternthat is indicative of the user’s performance. Moreover, the knowledgeserver 104 can determine that as a user’s grit increases over time, theperformance of the user, e.g., number of questions right divided bytotal number of questions for a task, also increases over time. As theknowledge server 104 calculates more grit scores, the knowledge server104 can continuously build a more accurate model that illustrates therelationship between grit and performance. Additionally, the knowledgeserver 104 can build models between other cognitive and behavioralmetrics and performance to determine if other direct correlations exist.

FIG. 5A is an illustration of example of anti-cheating identification.In some implementations, the knowledge server performs identityverification of the user writing with the writing implement. Theknowledge server can determine, in addition to verifying usercredentials, that the user writing with the writing implement is in factwho they say they are. Typically, one of the following methods can beused to perform identity verification: analyzing a photo or an image ofa user’s face, comparing a written signature to a signature on the backof a debit or credit card, and, a comparison through a fingerprintscanner, to name a few examples. The systems 100 and 101 can performidentity verification without using either of these typical methods, andrather, can perform identity verification by using just characteristicsof the user’s writing behavior, e.g., an amount of ink written by a userwith the writing implement 108.

For example, as illustrated in FIG. 5A, the receiving device can prompta series of characters, such as a phrase shown in box 502, for the userto write. The knowledge server, such as knowledge server 104, canprovide the phrase for the user to write over a network. The phrase canbe, for example, “The quick brown fox jumps over the lazy dog. Ineducation today, students are assessed, too simplistically, on absolutescores. Since students learn at different levels and different speeds,this practice is outdated practice, inefficient and incomplete, leavingstudents either unchallenged or overburdened. To address this criticalneed for change in education, ANOTO introduced KAIT — a new andinnovative AI — powered platform focused on enabling personalizedlearning and improving learning efficiencies. Our goal is to disruptglobal education and improve efficiency for both teachers and students.”In response to providing the statement to the user, the user writes thisstatement in his/her own handwriting using the writing implement on thereceiving device. Then, the computing system, receiving device, and thewriting implement can perform processes similar to the processesdescribed with respect to stages (A) through (F) of systems 100 and 101.

Additionally, the knowledge server can extract, from the input data,features that indicate a characteristic of a writing behavior of theuser while writing the designated phrase, e.g., the amount of inkwritten by user for each letter and each word. Moreover, the knowledgeserver can measure the amount of ink for each letter and/or for eachword, which can correspond to a digital representation of anidentification of the user. Other users can also write the same phrase,as shown in later figures, to have their identity verified. Generally,for the exact same phrase, each user will write the phrase using adifferent amount of ink that is unique to that user. The various lettersizes, word sizes, font size, writing style, stroke lengths, andpressure applied to the writing implement by each user can contribute tothe unique ink amount, which can be used to identify a particular user.Thus, each user has a particular fingerprint in the manner in which theywrite. The digital ink amount represents the fingerprint unique to thatindividual.

As illustrated in FIG. 5A, the graph 504 illustrates the amount of inkused by Alex for writing the phrase shown in box 502. For example, whenAlex wrote the phrase of “The quick brown fox jumps over the lazy dog,”the knowledge server measured from the writing implement’s data that anink amount in row 1 of graph 504 includes [63.6, 74.56, 45.64, 68.05,45.44, 76.89, 22.4] values, which was created by Alex for thatparticular phrase. In some implementations, the knowledge server cansegment the phrase by various limiters. These limiters can include, forexample, spaces, words, letters, every set number of letters, and thelike. Other values are also shown in graph 504, which corresponds toAlex’s unique fingerprinting for writing this phrase.

In response to measuring the ink amount shown in the graph 504, theknowledge server can compare the ink amount for each letter and/or wordof the statement to one or more datasets of ink amounts for that userfor the same statement. The knowledge server can store in a database,e.g., the user profiles database or another database, templates of inkamounts written for each letter by each user that uses the system. Forexample, the knowledge server can store an ink amount for each letter ofthe alphabet, both capital and lowercase, and for every number 1 through100 for each user. The ink amounts can range from 0 to 100, with 1 beinga very little amount of ink and 99 being a large amount of ink. Althoughmore numbers exist than 1 to 100, the larger numbers can be identifiedby one or more combinations of the numbers listed between numbers 1through 100. The knowledge server may request each user write out everyletter (capital and lowercase) and numbers 1 through 100 with thewriting implement before being able to use this system. In this manner,the knowledge server can store a template for each letter and number foreach user.

Continuing with the example, the knowledge server can compare the inkamounts of [63.6, 74.56, 45.64], which can correspond either to “Thequick brown fox” phrase or to one or more letters from that phrase, toink amounts for similar words or letters written by Alex previously. Ifthe knowledge server determines that the comparison amount is within athreshold, such as within 10 units of ink, then the knowledge server candetermine that the user writing with the writing implement is in factAlex based upon an additional comparison that Alex’s providedcredentials matches to the stored credentials. In some implementations,the credentials may be compared and verified before Alex is allowed tobegin writing with the writing implement. In some implementations, ifthe knowledge server determines that the comparison amount is outsidethe threshold, such as 10 units of ink, then the knowledge server candetermine that the user writing with the writing implement is not Alex,and can transmit a notification to a supervisor to indicate that Alex ischeating or another user not Alex is signed in and acting as Alex. Insome implementations, if the knowledge server determines that thecomparison is within the threshold, then the knowledge server cancompare Alex’s provided credentials to the stored credentials. If thesecredentials do not match, the knowledge server can indicate to Alex thatthe provided credentials are incorrect or indicate that the alleged Alexuser is in fact another user. In some implementations, the comparison ofeach ink amount, for letters or words, must be below the threshold valuefor a match to occur. If one or a subset of ink amounts are outside thethreshold, then a match does not exist, as further described below.

In some implementations, each letter or each word from the ink amountsmust be within the threshold for the knowledge server to indicate thatthe user writing with the writing implement is in fact who they claim tobe. If an ink amount associated with a letter or word is outside thethreshold, e.g., the letter “e” in the word “the” from the phrase 502has 21.2 ink, whereas the stored value of the letter “e” for Alex is 9ink, then the knowledge server can deem that the user writing the phrase502 is not Alex. However, other threshold values may be used to improvethe accuracy of the system.

FIG. 5B is an illustration of an example of anti-cheatingidentification. FIG. 5B is similar to FIG. 5A, except that the userAngela wrote the phrase instead of Alex. Angela wrote the same phrase asAlex, yet as illustrated in FIG. 5B, Angela writes with a differentstyle and font, and the knowledge server can determine that Angelawrites the same phrase 508 with a different ink amount for each letterand word. The ink amounts exhibited by Angela are illustrated in graph506. Clearly, graph 506 exhibits different ink amounts from the inkamounts exhibited in graph 504. Thus, the ink amount illustrated ingraph 506 is unique to Angela and can be used as a fingerprint foridentifying Angela.

FIG. 5C is an illustration of an example of anti-cheatingidentification. FIG. 5C is similar to FIGS. 5A and 5B, except that theuser Aziz wrote the same phrase instead of Alex and Angels. Aziz wrotethe same phase as Alex and Angela, yet as illustrated in FIG. 5C, Azizwrites with a different style and font, and the knowledge server candetermine that Aziz writes the same phrase 512 with a different inkamount for each letter and word. The ink amounts exhibited by Aziz areillustrated in graph 510. Clearly, graph 510 exhibits different inkamounts from the ink amounts exhibited in both graphs 504 and 506. Thus,the ink amount illustrated in graph 510 is unique to Aziz and can beused as a fingerprint for identifying Aziz.

FIG. 6A is an illustration of an example plot 600 showing individualheuristics for different letters for different users that use thewriting implement. Not only does each user write with a unique amount,but also each user writes with a unique letter size. For example, theplots 600 illustrate three different plots for Jood, Alex, andStephanie. The top plot illustrates the capital and lowercase lettersizes for Jood, the middle plot illustrates the capital and lowercaseletter sizes for Alex, and the lower plot illustrates the capital andlowercase letter sizes for Stephanie. As illustrated, each user writesletters in different sizes. The sizes can be measured, for example, inheight and length in centimeters, or a similar dimension.

In some implementations, to create these individual heuristics, theknowledge server may provide an indication to each user through acorresponding client device to write each letter of the alphabet incapital and lowercase and to write various numbers. The knowledge servercan obtain data of each user writing the letters of the alphabet and thenumbers and store ink amounts, sizes, other extracted features for eachof these written letters/numbers, and generated cognitive and behavioralmetrics for that user in a template, e.g., such as features and scoresdetermined during stages (A) through (F) in systems 100 and 101. Thetemplate can indicate writing styles, fonts, and other extractedfeatures that define a user’s writing style. This knowledge server mayask each user to write these letters and numbers with the writingimplement before being assigned a task. For example, the knowledgeserver can ask the user to write letters A-Z in lowercase and capitaland numbers 1- 100. The knowledge server can then use these templatesfor each user later when performing anti-cheating identification forthat user.

By storing and tracking letter and number sizes for each user, theknowledge server can develop and extract heuristics about eachparticular user because letter and number sizes are also unique to theusers. Heuristic analyses can be used to help identify a user and assistin identifying a user profile. Thus, when a user writes a particularword, the knowledge server can identify the amount of ink, the size ofeach letter and/or number, number of strokes, and other importantfeatures for that word, to identify that user because these features areunique to that user. Moreover, the knowledge server can look atheuristics from past data for that user, to develop an estimate of anexpected amount of ink to be used. This comparison can be used toidentify a particular individual and determine whether a particularindividual writing with the writing implement is cheating. For example,if a user John regularly writes the letter ‘t’ with the size of 12 andan ink amount of 20, then the knowledge server can determine that if theuser John writes a letter ‘t’ with the size of 12.2 and an ink amount of19.5, then the knowledge server can determine with a high degree ofconfidence that John is who he says he is.

FIG. 6B is an illustration of an example plot 601 showing individualheuristics for different letters for different users that use thewriting implement. The plots 601 illustrate three different plots forJood, Alex, and Stephanie. The top plot illustrates the Jood’s averagespeeds for writing capital and lowercase letters, the middle plotillustrates Alex’s average speeds for writing capital and lowercaseletters, and the lower plot illustrates Stephanie’s average speeds forwriting capital and lowercase letters. Similar to plots 600, each userwrites at different speeds for the various letters. The knowledge servercan similarly track the writing speeds for each user. These speeds canbe relied upon at a later point, when each user performs a task, and theknowledge server can use heuristics to develop an estimate of anexpected writing speed for the user.

FIG. 7 is a flow diagram that illustrates an example of a process 700for determining cognitive and behavioral metrics of an individual usinga writing implement. The knowledge server 104 or the computing system102 can perform the process 700, for example.

In the process 700, the knowledge server obtains data from a writingimplement, the data indicative of a user performing a task by writingwith the writing implement against a receiving device (702). As the userwrites with the writing implement against the receiving device or afterthe user completes writing with the writing implement against thereceiving device for a task, the writing implement can provide raw dataor biometric data to the knowledge server. In some implementations, boththe writing implement and the receiving device may provide the biometricdata to the knowledge server. The receiving device can be, for example,at least one of paper, smart paper, a tablet or a computer screen.

The data, obtained by the knowledge server, may include environmentaldata that include media illustrating (i) recorded video or images of theuser performing the task with the writing implement, (ii) an environmentin which the user is perform the task with the writing implement, and(iii) audio data recorded in the environment indicative of audiblenoises while the user is performing the task with the writing implement.Additionally, the data, obtained by the knowledge server, can alsoinclude test question data. The test question data can describe (i) oneor more questions posed to the user to be answered with the writingimplement and receiving device and (ii) a correct answer to each of theone or more questions. For example, the user can be posed to answer analgebra problem in a multiple choice question. The data, obtained by theknowledge server, can also include input data that describes (i) dataidentifying the writing implement and (ii) credentials of the userwriting with the writing implement. When the knowledge server generatesrecommendations tailored to improve a performance of the user, theknowledge server can generate the recommendations based on the testquestion data, the input data, and the environmental data.

The knowledge server extracts features from the obtained data, thefeatures indicating characteristics of a writing behavior of the userwhile the user performs the task with the writing implement (704). Forexample, the knowledge server can extract spatial features, temporalfeatures, pressure features, textual features, and tilt features. Thespatial features can describe spatial features of handwritten textproduced by the user with the writing implement while performing thetask. The temporal features can describe temporal elements ofhandwritten movements produced by the user with the writing implementwhile performing the task. The pressure features can describe a pressurerecorded between a tip of the writing implement and the receiving devicewhile the user performs the task. The textual features can describe thecharacteristics of the written letters and numbers with the writingimplement, such as optical character recognition data of the textualfeatures. The tilt features can describe one or more tilting angles ofthe writing implement while the user performs the task.

The knowledge server determines, based on at least some of the extractedfeatures from the obtained data, metrics that reflect characteristics ofthe user (706). For example, the knowledge server can determinecognitive metrics that include a number sense metric, a memoryvisualization metric, a reasoning metric, and an applicable connectionsmetric based on at least some of the extracted features. Additionally,the knowledge server can determine behavioral metrics that include atest taking skill metric, a grit metric, a concentration metric, and aquestion comprehension metric based on at least some of the extractedfeatures.

The knowledge server generates a user profile for the user based on theextracted features, the determined metrics, or both (708). For example,the user profile can be a generated data structure that includes each ofthe cognitive metrics and the behavioral metrics for a user. The datastructure can be, for example, a tuple, a data struct, a vector, anarray, and the like. The knowledge server can associate receivedcredentials with the generated user profile. The generated user profilecan also store the extracted features. Additionally, the generated userprofile can include data that monitors a user’s performance over time,such as sets of cognitive and behavioral metrics.

Based on the features indicating characteristics of the writing behaviorof the user, the knowledge server modifies the generated user profile asa performance of the user changes over time (710). For example, as theknowledge server generates new cognitive and behavioral metrics for auser each time the user performs a task, the knowledge server can updatethese metrics in a user’s profile. The knowledge server may (i)recalculate the cognitive and behavioral metrics based on a movingaverage over time, (ii) recalculate the cognitive and behavioral metricson a periodic basis, and (iii) erase the previously calculated cognitiveand behavioral metrics and save the newly calculated cognitive andbehavioral metrics with the user’s profile. In some implementations, theknowledge server can store each of the previously calculated metrics sothat the user or a supervisor of the user can monitor the user’sperformance progress.

For example, the user or the supervisor can monitor whether the user’sperformance progress is slowing, acceleration, or remaining constantover time. If the user’s grit, number sense, and test taking metrics,for example, all appear to be increasing over time, then the user’sperformance may be increasing. In another example, if the user’s grit,number sense, and test taking metric, for example, all appear to bedecreasing over time, then the user’s performance may be decreasing.More or less of the cognitive and behavioral metrics than thosedescribed in the previous examples may be used to determine if a user’sperformance is increasing or decreasing over time.

The knowledge server provides recommendations to improve the performanceof the user associated with the generated user profile, therecommendations based on the determined metrics and the generatedprofile (712). The recommendations can also include an indication ofwhether a written response provided by the user is correct or incorrect,and the written response corresponds to the response written by the userperforming the specified task. The recommendations can also include oneor more of the cognitive and behavioral metrics, such as the gritmetric, and one or more corrective actions to improve the performance ofthe user. The one or more corrective actions can include, for example,one of (i) instructions to perform additional tasks of a similar type tothe task, (ii) instructions to reduce an amount of noise in anenvironment, and (iii) medical information of the user.

Based on a trend of performance of the user, the knowledge server cangenerate recommendations to provide to the user or a supervisor of theuser. For example, the knowledge server may instruct the user to performadditional problem types to improve certain cognitive and behavioralmetrics; instruct the user to adjust his environment if the knowledgeserver determines that some of the environmental aspects appear to beaffecting the user’s performance on tasks, e.g., such as loud music, aloud television, or loud individuals in the room. Additionally, theknowledge server can instruct the user or a supervisor withrecommendations, such as, “Practice math in head,” “Practice mathproblems on receiving device,” or “Practice problems in a timedenvironment, to improve certain metrics, such as number sense, grit, andtest taking.

FIG. 8 is a flow diagram that illustrates an example of a process 800for validating an identity of a user. The knowledge server 104 or thecomputing system 102 can perform the process 800, for example.

In the process 800, the knowledge server obtains data from a writingimplement, the data indicative of a user writing a series of characterswith the writing implement on a receiving device (802). As the userwrites with the writing implement against the receiving device or afterthe user completes writing with the writing implement against thereceiving device for a task, the writing implement can provide data tothe knowledge server.

The user can write a series of characters, e.g., letters, numbers, orother symbols, designed to match an instructional task. The task canindicate that the user write a phrase or one or more sentences. Forexample, the knowledge server may provide an indication that the userneeds to write each letter of the alphabet in capital and lowercaseletters along with numbers 1-100 on the receiving device with thewriting implement. The knowledge server can also request the user writein different languages, write different symbols, and in differentwriting styles, such as cursive or italics. The knowledge server mayrequest the user write the letters, numbers, and symbols to build aprofile of characters for the user. Then, at a different time, theknowledge server may request the user write a different phrase,sentence, sentences, or paragraphs to ensure the user is in fact notcheating. The knowledge server can also perform user or identityauthentication when the user performs other tasks, such as math problemsor reading comprehension type problems.

In some implementations, both the writing implement and the receivingdevice may provide the data to the knowledge server. The receivingdevice can be, for example, at least one of paper, smart paper, a tabletor a computer screen. The data obtained from the writing implement in802 is similar to the data obtained by the knowledge server in 702.

The knowledge server extracts features from the obtained data, thefeatures indicating characteristics of a writing behavior of the userwhile writing the series of characters (804). 804 is similar to 704. Forexample, the knowledge server can extract spatial features, temporalfeatures, pressure features, textual features, and tilt features. Thespatial features can describe spatial features of handwritten textproduced by the user with the writing implement while writing the seriesof characters. The temporal features can describe temporal elements ofhandwritten movements produced by the user with the writing implementwhile writing the series of characters. The pressure features candescribe a pressure recorded between a tip of the writing implement andthe receiving device while the user writes the series of characters. Thetextual features can describe the characteristics of the written lettersand numbers with the writing implement, such as optical characterrecognition data of the textual features. The tilt features can describeone or more tilting angles of the writing implement while the userwrites the series of characters.

The knowledge server compares a characteristic dataset based on theextracted features to one or more stored characteristic datasets for thesame series of characters (806). For example, the knowledge server canmeasure an amount of ink generated by the writing implement for eachcharacter in the written series of characters. The characteristicdataset can include data based on the measured amount of ink for eachcharacter. Thus, the knowledge server can compare the characteristicdataset the one or more stored datasets by comparing the amount of inkfor each character written by the user to the one or more storeddatasets using a heuristic analysis. For example, if the user wrote thewords “DOG PLAY”, the knowledge server would measure an amount of inkfor each character written by the user in the words “DOG PLAY” andcompare that measured amount of ink for each written character to storedamounts of ink written previously for the same letters, e.g., “DOGPLAY”. The knowledge server can use a heuristic analysis to perform thecomparison between written ink amounts for certain letters to stored inkamounts of the same previously written letters.

The knowledge server, in response to identifying a stored characteristicdataset that matches the characteristic dataset based on the extractedfeatures, can validate an identity of the user (808). The knowledgeserver can validate an identity of the user by determining whethercredentials of the user, e.g., username and password, match to anidentification of user associated with the identified stored dataset.For example, the knowledge server can determine whether the providedcredentials matches stored credentials associated with the identifiedstored dataset. If a match exists, the knowledge server can determinethat the user is not cheating, e.g., the user is who he claims to be.

However, if the knowledge server cannot identify a stored characteristicdataset that matches to the written series of characters, the knowledgeserver can provide an indication to the user or to the user’s supervisorindicating that the user is cheating, and the user is not who he claimsto be. For example, the user who claims to be “John” is not “John.” Ifthe knowledge server determines that the user’s writing actually matchesto another user’s stored dataset, then the knowledge server candetermine and provide an indication to a supervisor that (i) the user ischeating and (ii) the user is in fact someone another user. For example,the user who claims to be “John” is in fact “Mark,” and in fact, theuser “John” is cheating.

Various implementations of the systems and techniques described here canbe realized in digital electronic circuitry, integrated circuitry,specially designed ASICs (application specific integrated circuits),computer hardware, firmware, software, and/or combinations thereof.These various implementations can include implementation in one or morecomputer programs that are executable and/or interpretable on aprogrammable system including at least one programmable processor, whichmay be special or general purpose, coupled to receive data andinstructions from, and to transmit data and instructions to, a storagesystem, at least one input device, and at least one output device.

These computer programs (also known as programs, software, softwareapplications or code) include machine instructions for a programmableprocessor, and can be implemented in a high-level procedural and/orobject-oriented programming language, and/or in assembly/machinelanguage. As used herein, the terms machine-readable medium andcomputer-readable medium refer to any computer program product,apparatus and/or device (e.g., magnetic discs, optical disks, memory,Programmable Logic Devices (PLDs)) used to provide machine instructionsand/or data to a programmable processor, including a machine-readablemedium that receives machine instructions as a machine-readable signal.The term machine-readable signal refers to any signal used to providemachine instructions and/or data to a programmable processor.

To provide for interaction with a user, the systems and techniquesdescribed here can be implemented on a computer having a display device(e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor)for displaying information to the user and a keyboard and a pointingdevice (e.g., a mouse or a trackball) by which the user can provideinput to the computer. Other kinds of devices can be used to provide forinteraction with a user as well; for example, feedback provided to theuser can be any form of sensory feedback (e.g., visual feedback,auditory feedback, or tactile feedback); and input from the user can bereceived in any form, including acoustic, speech, or tactile input.

The systems and techniques described here can be implemented in acomputing system that includes a back end component (e.g., as a dataserver), or that includes a middleware component (e.g., an applicationserver), or that includes a front end component (e.g., a client computerhaving a graphical user interface or a Web browser through which a usercan interact with an implementation of the systems and techniquesdescribed here), or any combination of such back end, middleware, orfront end components. The components of the system can be interconnectedby any form or medium of digital data communication (e.g., acommunication network). Examples of communication networks include alocal area network (LAN), a wide area network (WAN), and the Internet.

The computing system can include clients and servers. A client andserver are generally remote from each other and typically interactthrough a communication network. The relationship of client and serverarises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other.

Although a few implementations have been described in detail above,other modifications are possible. For example, while a clientapplication is described as accessing the delegate(s), in otherimplementations the delegate(s) may be employed by other applicationsimplemented by one or more processors, such as an application executingon one or more servers. In addition, the logic flows depicted in thefigures do not require the particular order shown, or sequential order,to achieve desirable results. In addition, other actions may beprovided, or actions may be eliminated, from the described flows, andother components may be added to, or removed from, the describedsystems. Accordingly, other implementations are within the scope of thefollowing claims.

While this specification contains many specific implementation details,these should not be construed as limitations on the scope of anyinvention or of what may be claimed, but rather as descriptions offeatures that may be specific to particular embodiments of particularinventions. Certain features that are described in this specification inthe context of separate embodiments can also be implemented incombination in a single embodiment. Conversely, various features thatare described in the context of a single embodiment can also beimplemented in multiple embodiments separately or in any suitablesubcombination. Moreover, although features may be described above asacting in certain combinations and even initially claimed as such, oneor more features from a claimed combination can in some cases be excisedfrom the combination, and the claimed combination may be directed to asubcombination or variation of a subcombination.

Similarly, while operations are depicted in the drawings in a particularorder, this should not be understood as requiring that such operationsbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. In certain circumstances, multitasking and parallel processingmay be advantageous. Moreover, the separation of various system modulesand components in the embodiments described above should not beunderstood as requiring such separation in all embodiments, and itshould be understood that the described program components and systemscan generally be integrated together in a single software product orpackaged into multiple software products.

Particular embodiments of the subject matter have been described. Otherembodiments are within the scope of the following claims. For example,the actions recited in the claims can be performed in a different orderand still achieve desirable results. As one example, the processesdepicted in the accompanying figures do not necessarily require theparticular order shown, or sequential order, to achieve desirableresults. In certain implementations, multitasking and parallelprocessing may be advantageous.

What is claimed is:
 1. (canceled)
 2. A computer-implemented methodcomprising: obtaining data from a writing implement, the data indicativeof a first user writing a series of characters with the writingimplement on a receiving device; extracting features from the obtaineddata, the features indicating characteristics of a writing behavior ofthe user while writing the series of characters; comparing acharacteristic data set based on the extracted features to one or morestored characteristic datasets for the same series of characters,wherein each stored characteristic dataset of the one or more storedcharacteristic datasets is associated with a respective user; andidentifying a stored characteristic dataset that matches thecharacteristic dataset based on the extracted features; and in responseto identifying the stored characteristic dataset, validating an identityof the first user.
 3. The computer-implemented method of claim 2,wherein the receiving device comprises at least one of paper, smartpaper, a tablet, or a computer screen.
 4. The computer-implementedmethod of claim 2, wherein the one or more stored characteristicdatasets associated with the respective user comprises the one or morestored characteristic datasets written by the respective user using arespective writing implement.
 5. The computer-implemented method ofclaim 2, comprising: obtaining, by the one or more computers, one ormore of: test question data that describe (i) one or more questionsposed to the first user to be answered with the writing implement andthe receiving device, (ii) a correct answer to each of the one or morequestions, and (iii) an answer rubric, and input data that describe (i)data identifying the writing implement and (ii) credentials of the firstuser writing with the writing implement.
 6. The computer-implementedmethod of claim 2, wherein extracting the features from the obtaineddata comprises one or more of: extracting, by the one or more computers,temporal features that describe temporal elements of handwrittenmovements produced by the first user with the writing implement whileperforming a task; extracting, by the one or more computers, pressurefeatures that describe a pressure recorded between a tip of the writingimplement and the receiving device while the first user performs thetask; and extracting, by the one or more computers, textual featuresthat describe characteristics of written letters and numbers with thewriting implement, wherein the textual features also comprise timestamps associated with each of handwritten text.
 7. Thecomputer-implemented method of claim 6, wherein comparing thecharacteristic data set based on the extracted features to the one ormore stored characteristic data sets for the same series of charactersfurther comprises: measuring an amount of ink for each of one or morecharacters of the series of characters written by the writing implement;identifying one or more characters of the series of characters writtenby the writing implement; and identifying one or more storedcharacteristic datasets using (i) the amount of ink for each of the oneor more characters and (ii) the identification of each of the one ormore characters of the series of characters, the identifying comprisingcomparing the measured amount of ink for each of the one or morecharacters of the series of characters written by the writing implementto an amount of ink for each character in the one or more storedcharacteristic datasets.
 8. The computer-implemented method of claim 7,wherein identifying the stored characteristic dataset that matches thecharacteristic dataset based on the extracted features furthercomprises: determining whether the comparison between the measured inkamounts for each of the one or more characters of the series ofcharacters and the amount of ink for each of the one or more charactersin the one or more stored characteristic datasets satisfies a thresholdvalue; in response to determining the comparison satisfies the thresholdvalue for at least one of the stored characteristic datasets,identifying a corresponding stored characteristic dataset as the datasetfor identity validation.
 9. The computer-implemented method of claim 8,wherein validating the identity of the first user further comprises: inresponse to identifying the corresponding stored characteristic datasetas the dataset for the identity validation, the method comprises:identifying the respective user associated with the storedcharacteristic dataset; obtaining, from a database, first usercredentials of the first user writing with the writing implement;obtaining second user credentials of the respective user w; andcomparing the first user credentials with the second user credentials todetermine whether a match exists.
 10. The computer-implemented method ofclaim 9, wherein comparing the first user credentials with the seconduser credentials further comprises: in response to determining the firstuser credentials do not match to the second user credentials,determining that an identification of the first user does not match toan identification of the respective user; and transmitting data to asupervisor indicating that the first user is cheating during the firstuser’s writing the series of characters with the writing implement onthe receiving device.
 11. The computer-implemented method of claim 9,wherein comparing the first user credentials with the second usercredentials further comprises: in response to determining the first usercredentials do match to the second user credentials, determining anidentification of the first user does match to an identification of therespective user; and validating the identity of the first user based onthe first user’s writing the series of characters with the writingimplement on the receiving device.
 12. The computer-implemented methodof claim 8, further comprising: in response to determining thecomparison does not satisfy the threshold value for at least one of thestored characteristic datasets, determining an identification of thefirst user is unknown.
 13. A system comprising: one or more computersand one or more storage devices storing instructions that are operable,when executed by the one or more computers, to cause the one or morecomputers to perform operations comprising: obtaining data from awriting implement, the data indicative of a first user writing a seriesof characters with the writing implement on a receiving device;extracting features from the obtained data, the features indicatingcharacteristics of a writing behavior of the user while writing theseries of characters; comparing a characteristic data set based on theextracted features to one or more stored characteristic datasets for thesame series of characters, wherein each stored characteristic dataset ofthe one or more stored characteristic datasets is associated with arespective user; and identifying a stored characteristic dataset thatmatches the characteristic dataset based on the extracted features; andin response to identifying the stored characteristic dataset, validatingan identity of the first user.
 14. The system of claim 13, wherein thereceiving device comprises at least one of paper, smart paper, a tablet,or a computer screen.
 15. The system of claim 13, wherein the one ormore stored characteristic datasets associated with the respective usercomprises the one or more stored characteristic datasets written by therespective user using a respective writing implement.
 16. The system ofclaim 13, comprising: obtaining, by the one or more computers, one ormore of: test question data that describe (i) one or more questionsposed to the first user to be answered with the writing implement andthe receiving device, (ii) a correct answer to each of the one or morequestions, and (iii) an answer rubric, and input data that describe (i)data identifying the writing implement and (ii) credentials of the firstuser writing with the writing implement.
 17. The system of claim 13,wherein extracting the features from the obtained data comprises one ormore of: extracting, by the one or more computers, temporal featuresthat describe temporal elements of handwritten movements produced by thefirst user with the writing implement while performing a task;extracting, by the one or more computers, pressure features thatdescribe a pressure recorded between a tip of the writing implement andthe receiving device while the first user performs the task; andextracting, by the one or more computers, textual features that describecharacteristics of written letters and numbers with the writingimplement, wherein the textual features also comprise time stampsassociated with each of handwritten text.
 18. The system of claim 17,wherein comparing the characteristic data set based on the extractedfeatures to the one or more stored characteristic data sets for the sameseries of characters further comprises: measuring an amount of ink foreach of one or more characters of the series of characters written bythe writing implement; identifying one or more characters of the seriesof characters written by the writing implement; and identifying one ormore stored characteristic datasets using (i) the amount of ink for eachof the one or more characters and (ii) the identification of each of theone or more characters of the series of characters, the identifyingcomprising comparing the measured amount of ink for each of the one ormore characters of the series of characters written by the writingimplement to an amount of ink for each character in the one or morestored characteristic datasets.
 19. The system of claim 18, whereinidentifying the stored characteristic dataset that matches thecharacteristic dataset based on the extracted features furthercomprises: determining whether the comparison between the measured inkamounts for each of the one or more characters of the series ofcharacters and the amount of ink for each of the one or more charactersin the one or more stored characteristic datasets satisfies a thresholdvalue; in response to determining the comparison satisfies the thresholdvalue for at least one of the stored characteristic datasets,identifying a corresponding stored characteristic dataset as the datasetfor identity validation.
 20. The system of claim 19, wherein validatingthe identity of the first user further comprises: in response toidentifying the corresponding stored characteristic dataset as thedataset for the identity validation, the system comprises: identifyingthe respective user associated with the stored characteristic dataset;obtaining, from a database, first user credentials of the first userwriting with the writing implement; obtaining second user credentials ofthe respective user w; and comparing the first user credentials with thesecond user credentials to determine whether a match exists.
 21. Anon-transitory computer-readable medium storing software comprisinginstructions executable by one or more computers which, upon suchexecution, cause the one or more computers to perform operationscomprising: obtaining data from a writing implement, the data indicativeof a first user writing a series of characters with the writingimplement on a receiving device; extracting features from the obtaineddata, the features indicating characteristics of a writing behavior ofthe user while writing the series of characters; comparing acharacteristic data set based on the extracted features to one or morestored characteristic datasets for the same series of characters,wherein each stored characteristic dataset of the one or more storedcharacteristic datasets is associated with a respective user; andidentifying a stored characteristic dataset that matches thecharacteristic dataset based on the extracted features; and in responseto identifying the stored characteristic dataset, validating an identityof the first user.